Information processing system and information processing apparatus

ABSTRACT

An information processing system and apparatus which have a plurality of actuators, a plurality of sensors, a signal processing unit, a control unit, and an integrating unit. Various types of signals are exchanged as necessary via a network. Actuators and sensors share processing through collaboration so that individual functionality is raised. The present invention can be achieved through a SHARN system (Sensor, Human, Actuator, Real world, and Network) which can be applied to surgery, Personal Digital Assistant design, and the like, for example.

TECHNICAL FIELD

The present invention relates to an information processing system and aninformation processing device, and particularly relates to aninformation processing system and an information processing deviceenabling highly functional processing by, for example, multiple actionmeans acting in response to operations of multiple users, or multipledetecting means for detecting the real world in which the multipleacting means act, cooperating to share the load of processing.

BACKGROUND ART

For example, with multiple processors (here, something which performssome sort of processing), performing a certain processing by sharing theload in parallel enables the processing speed to be increased by theorder of the number of the multiple processors as compared to oneprocessor performing the processing, according to simple arithmetic.

Now, regardless of whether the processing is carried out with oneprocessor or the processing is shared among multiple processors, theobtained processing results are no different. Further, the functions ofeach processor in a case of performing processing by sharing betweenmultiple processors are no different from the case of performing theprocessing with one processor alone.

That is to say, conventionally, however many processors are collected,only the overall processing speed increases according to the number ofprocessors, and the functions of each processor do not change.

Now, it is known by experience that in the event that multiple humanscome together to perform a task, individuals cooperating (collaborating)with each other enables each individual to perform work which exceedsthe normal capabilities thereof.

Accordingly, it can be thought that in a case wherein multipleprocessors are collected, sharing the processing between the processorsthrough collaboration would allow the functions of each processor to beincreased, so as to exhibit overall processing functions exceeding theorder of the number of the processors, and moreover to attach addedvalue to the overall processing results.

Also, it can be thought that in the event of multiple humans gatheringto perform a task, rather than gathering humans with the same attributessuch as disposition, mindset, and sensitivity, gathering humans withdifferent attributes will mutually stimulate one another, and furtherself-development even more.

Accordingly, it can be thought that with processors as well, rather thangathering multiple processors having the same functions, gatheringprocessors having mutually different functions, and having theprocessors share the processing through collaboration, would increasethe functions of each processor.

DISCLOSURE OF INVENTION

The present invention has been made in light of such a situation, and itis an object thereof to improve the functions of each by sharingprocessing among many through collaboration.

The information processing system according to the present inventioncomprises: a plurality of acting means which act corresponding tooperations by a plurality of users; a plurality of detecting means fordetecting the real world where the plurality of acting means act; andcommunication means for exchanging information between the plurality ofacting means and the plurality of detecting means; wherein informationis exchanged by the communication means and the plurality of actingmeans and the plurality of detecting means share processing throughcollaboration, whereby the plurality of acting means or the plurality ofdetecting means are more highly functional as compared to operatingindividually.

With a first information processing method according to the presentinvention, a plurality of acting means act corresponding to operationsby a plurality of users; a plurality of detecting means detect the realworld where the plurality of acting means act; and communication meansexchange information between the plurality of acting means and theplurality of detecting means; wherein information is exchanged by thecommunication means and the plurality of acting means and the pluralityof detecting means share processing through collaboration, whereby theplurality of acting means or the plurality of detecting means are morehighly functional as compared to operating individually.

A first computer-readable program according to the present inventioncontrols a computer such that a plurality of acting means actcorresponding to operations by a plurality of users; a plurality ofdetecting means detect the real world where the plurality of actingmeans act; and communication means exchange information between theplurality of acting means and the plurality of detecting means; whereininformation is exchanged by the communication means and the plurality ofacting means and the plurality of detecting means share processingthrough collaboration, whereby the plurality of acting means or theplurality of detecting means are more highly functional as compared tooperating individually.

A program of a first recording medium according to the present inventioncontrols a computer such that a plurality of acting means actcorresponding to operations by a plurality of users; a plurality ofdetecting means detect the real world where the plurality of actingmeans act; and communication means exchange information between theplurality of acting means and the plurality of detecting means; whereininformation is exchanged by the communication means and the plurality ofacting means and the plurality of detecting means share processingthrough collaboration, whereby the plurality of acting means or theplurality of detecting means are more highly functional as compared tooperating individually.

The information processing device according to the present inventioncomprises: acting means which act corresponding to operations by users;detecting means for detecting the real world where the acting means act;and communication means for exchanging information between other actingmeans and other detecting means; wherein information is exchanged by thecommunication means and the plurality of acting means and the pluralityof detecting means share processing with other acting means anddetecting means through collaboration, whereby the plurality of actingmeans or the plurality of detecting means are more highly functional ascompared to operating individually.

With a second information processing method according to the presentinvention, acting means act corresponding to operations by a pluralityof users; detecting means detect the real world where the acting meansact; and information is exchanged between the other acting means and theother detecting means; wherein the acting means and the detecting meansshare processing with other acting means and detecting means throughcollaboration, thereby becoming more highly functional as compared tooperating individually.

A second computer-readable program according to the present inventioncontrols a computer such that acting means act corresponding tooperations by a plurality of users; detecting means detect the realworld where the acting means act; and information is exchanged betweenthe acting means and the detecting means; wherein the acting means andthe detecting means share processing with other acting means anddetecting means through collaboration, thereby becoming more highlyfunctional as compared to operating individually.

A program of a second storage medium according to the present inventioncontrols a computer such that acting means act corresponding tooperations by a plurality of users; detecting means detect the realworld where the acting means act; and information is exchanged betweenthe acting means and the detecting means; wherein the acting means andthe detecting means share processing with other acting means anddetecting means through collaboration, thereby becoming more highlyfunctional as compared to operating individually.

With the information processing system, first information processingmethod, first program, and program of first recording medium, accordingto the present invention, multiple acting means or multiple detectingmeans share processing through collaboration, so the functionality ofthe multiple acting means or multiple detecting means is raised beyondthat of operating individually.

With the information processing device, second information processingmethod, second program and program of second recording medium, accordingto the present invention, acting means or detecting means shareprocessing through collaboration with other acting means or detectingmeans, so the functionality is raised beyond that of operatingindividually.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of anembodiment of a SHARN system to which the present invention has beenapplied.

FIG. 2 is a block diagram illustrating a configuration example ofanother embodiment of the SHARN system to which the present inventionhas been applied.

FIG. 3 is a block diagram illustrating a configuration example of anembodiment of a SHARN surgery system to which the SHARN system has beenapplied.

FIG. 4A is a perspective view illustrating an external configurationexample of a slave manipulator 61.

FIG. 4B is a perspective view illustrating an external configurationexample of a master manipulator 62.

FIG. 5 is a diagram describing roll rotation, pitch rotation and yawrotation.

FIG. 6A is a diagram illustrating a target path.

FIG. 6B is a diagram illustrating an operation path.

FIG. 6C is a diagram illustrating an averaged path.

FIG. 7 is a block diagram illustrating an electrical configurationexample of the slave manipulator 61.

FIG. 8 is a block diagram illustrating an electrical configurationexample of the master manipulator 62.

FIG. 9 is a block diagram illustrating an electrical configurationexample of a super doctor database unit 64.

FIG. 10 is a flowchart describing the processing of the slavemanipulator 61.

FIG. 11 is a flowchart describing the processing of the mastermanipulator 62.

FIG. 12 is a flowchart describing the processing of the super doctordatabase unit 64.

FIG. 13 is a diagram describing collaboration sharing in a SHARN surgerysystem.

FIG. 14 is a block diagram illustrating a configuration example of anembodiment of a SHARN design system to which the SHARN system has beenapplied.

FIG. 15 is a block diagram illustrating an electrical configurationexample of a pseudo slave manipulator 41.

FIG. 16 is a block diagram illustrating an electrical configurationexample of a master manipulator 42.

FIG. 17 is a block diagram illustrating an electrical configurationexample of a super designer database unit 44.

FIG. 18A is a diagram describing design information.

FIG. 18B is a diagram describing design information.

FIG. 19 is a flowchart describing processing of the pseudo slavemanipulator 41.

FIG. 20 is a flowchart describing processing of the master manipulator42.

FIG. 21 is a flowchart describing processing of the super designerdatabase unit 44 (control processing).

FIG. 22 is a flowchart describing processing of the super designerdatabase unit 44 (updating processing).

FIG. 23 is a diagram describing collaboration sharing in a SHARN designsystem.

FIG. 24 is a block diagram illustrating a configuration example of anembodiment of a SHARN moving entity system to which the SHARN system hasbeen applied.

FIG. 25 is a diagram describing the overview of the functions of theSHARN moving entity system.

FIG. 26 is a block diagram illustrating an electrical configurationexample of a moving entity 101 _(k).

FIG. 27 is a perspective view illustrating a configuration example of animage-taking device 111.

FIG. 28 is a diagram illustrating all-directional image data.

FIG. 29 is a diagram describing a hyperboloid making up a condensingunit 202.

FIG. 30 is a diagram for describing the positional relation between thecondensing unit 202 and an image-taking unit 203.

FIG. 31 is a diagram illustrating all-directional data.

FIG. 32 is a diagram describing cropping from all-directional data.

FIG. 33 is a diagram for describing deformation in the circumferentialdirection.

FIG. 34 is a diagram for describing deformation in the radial direction.

FIG. 35 is a diagram for describing a method for removing deformation inthe radial direction.

FIG. 36 is a diagram for describing a method for removing deformation inthe radial direction.

FIG. 37A is a diagram illustrating all-directional image data beforeremoving deformation in the radial direction.

FIG. 37B is a diagram illustrating image data after removing deformationin the radial direction from the all-directional image data.

FIG. 38 is a diagram for describing how to obtain pixel values at apoint to be converted.

FIG. 39 is a block diagram illustrating an electrical configurationexample of a scaling unit 113.

FIG. 40 is a block diagram illustrating an electrical configurationexample of a learning device for learning tap coefficients.

FIG. 41 is a block diagram illustrating an electrical configurationexample of a coordinates computing unit 125.

FIG. 42 is a diagram for describing how to obtain image-taking points.

FIG. 43A is a diagram for describing how to obtain image-taking points.

FIG. 43B is a diagram for describing how to obtain image-taking points.

FIG. 44 is a diagram illustrating the storage format of an imagedatabase 114.

FIG. 45 is a block diagram illustrating an electrical configurationexample of a display image altering unit 119.

FIG. 46 is a flowchart describing the processing of a main movingentity.

FIG. 47 is a flowchart describing all-directional image acquiringprocessing.

FIG. 48 is a flowchart describing viewpoint-specified image acquiringprocessing.

FIG. 49 is a flowchart describing viewpoint-variable image acquiringprocessing.

FIG. 50 is a flowchart describing processing of a slave moving entity.

FIG. 51 is a diagram describing collaboration sharing in the SHARNmoving entity system.

FIG. 52 is a block diagram illustrating a configuration example of anembodiment of a computer to which the present invention has beenapplied.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 illustrates a configuration example of an embodiment of a SHARNsystem to which the present invention has been applied.

Now, “SHARN” of the SHARN system is an acronym of Sensor, Human,Actuator, real world (Real), and Network.

In FIG. 1, a network 1001 is a wide-area network to which asynthesizing/integrating unit 1002, driving control unit 1004, andsignal processing unit 1006 are mutually connected, and signals(information) are exchanged between the synthesizing/integrating unit1002, driving control unit 1004, and signal processing unit 1006 viathis network 1001. Note that the network 1001 may be a line network, awireless network, or a combined network of line and wireless. Also thenetwork 1001 corresponds to the N for Network in “SHARN”.

The synthesizing/integrating unit 1002 receives operating signals outputfrom each of operating units 1003 ₁ through 1003 _(K), synthesizes orintegrates these, and transmits to the network 1001. Here, synthesizingmeans placing multiple operating signals in a state wherein theoperating signals cannot be separated, by weighting addition or thelike, for example. Also, integrating means combining the multipleoperating signals into one signal which can be separated into eachoperating signal, by multiplexing or the like, for example.

The operating unit 1003 _(k) (wherein k=1, 2, . . . K) is operated by auser U_(K), and operating signals corresponding to the operationsthereof are supplied to the synthesizing/integrating unit 1002. Notethat with the embodiment in FIG. 1 (as with the later-described FIG. 2),a K number, which is a plurality, of operating units 1003 ₁ through 1003_(K), are provided. The user U_(K) which operates the operating unit1003 _(K) corresponds to the H for Human in “SHARN”.

The driving control unit 1004 receives the signals supplied via thenetwork 1001, and controls driving of actuators 1005 ₁ through 1005_(M), based on these signals.

The actuator 1005 _(m) (wherein m=1, 2, . . . , M) operates according tocontrol from the driving control unit 1004, and accordingly acts uponobjects in the real word in a predetermined way. With the firstembodiment in FIG. 1 (as with the later-described FIG. 2), an M number,which is a plurality, of actuators 1005 ₁ through 1005 _(M), areprovided. Also, the actuator 1005 _(m) corresponds to the A for Actuatorin “SHARN”, and further, the real world in which the actuator 1005 _(m)acts corresponds to the R for real world (Real) in “SHARN”.

The signal processing unit 1006 receives detection signals from each ofsensors 1007 ₁ through 1007 _(N), subjects top necessary signalsprocessing and transmits to the network 1001.

The sensor 1007 _(n) (wherein n=1, 2, . . . , N) detects the state inthe real world where the actuator 1005 _(m) acts, and supplies detectingsignals representing the detected state to the signal processing unit1006. Note that with the first embodiment in FIG. 1 (as with thelater-described FIG. 2), an N number, which is a plurality, of sensors1007 ₁ through 1007 _(N), are provided. Also, the sensor 1007 _(m)corresponds to the S for Sensor in “SHARN”, and further, the real worldin which the sensor 1007 _(m) acts corresponds to the R for real world(Real) in “SHARN”.

Hereafter, the operating units 1003 ₁ through 1003 _(K) will be referredto operating unit 1003 unless there is a particular need to distinguishtherebetween. In the same way, the actuators 1005 ₁ through 1005 _(M)will be described as actuator 1005, and the sensors 1007 ₁ through 1007_(N) will be described as sensor 1007.

Next, FIG. 2 illustrates a configuration example of another embodimentof the SHARN system to which the present invention has been applied. Theportions in the drawing which correspond to the case in FIG. 1 aredenoted with the same reference numerals, and description thereof willbe omitted as appropriate in the following.

That is to say, with the first embodiment shown in FIG. 1, the operatingunit 1003 is connected to the network 1001 via thesynthesizing/integrating unit 1002, the actuator 1005 via the drivingcontrol unit 1004, and the sensor 1007 via the signal processing unit1006, respectively, but with the second embodiment shown in FIG. 2, theoperating unit 1003, actuator 1005, and sensor 1007 are directlyconnected to the network 1001.

Accordingly, with the second embodiment shown in FIG. 2, operatingsignals output from the operating unit 1003 are supplied to thesynthesizing/integrating unit 1002 via the network 1001. Also, thesignals for controlling driving of the actuator 1005 output from thedriving control unit 1004 are supplied to the actuator 1005 via thenetwork 1001, and detecting signals output from the sensor 1007 aresupplied to the signals processing unit 1006 via the network 1001.

With the SHARN system configured as described above, various signals(information) are exchanged as necessary between the operating unit1003, actuator 1005, and sensor 1007, through thesynthesizing/integrating unit 1002, driving control unit 1004, andsignal processing unit 1006, via the network 1001. Further, varioussignals are exchanged via the network 1001, through thesynthesizing/integrating unit 1002 among the operating units 1003 ₁through 1003 _(K), through the driving control unit 1004 among theactuators 1005 ₁ through 1005 _(M), and through the signal processingunit 1006 among the sensors 1007. Moreover, with the SHARN system, thefunctionality of each of the actuators 1005 _(m) and each of the sensors1007 _(n) improve by the actuators 1005 and sensors 1007 collaboratingto share processing. Further, as a result of the functionality of eachof the actuators 1005 _(m) and each of the sensors 1007 _(n) improving,the processing results of the overall system are such that have addedvalue added thereto.

Note that with the embodiments in FIG. 1 and FIG. 2, K which is thenumber of operating units 1003, M which is the number of actuators 1005,and N which is the number of sensors 1007, may be the same value, or maybe different values.

Next, FIG. 3 illustrates a configuration example of a SHARN surgerysystem as a surgery system to which the SHARN system has been applied.

This SHARN surgery system is a manipulator system for surgery,configured of a slave manipulator 61, master manipulators 62 ₁ through62 _(K), a network 63, and a super doctor database unit 64.

The slave manipulator 61 acts based upon signals supplied via thenetwork 63, and performs endermic surgery or the like on an animal suchas a dog or the like, for example, which is the subject of surgery.Here, the slave manipulator 61 performs endermic surgery on an animal,but other arrangements may include the performing trimming or the likeon the animal.

The master manipulator 62 _(k) acts in response to operations of a userU_(k) which is a veterinarian for example, and information suppliedthrough the network 63 from the super doctor database 64. Also, themaster manipulator 62 _(k) detects the state of the actions, andtransmits the results of detection to the super doctor database 64 viathe network 63.

Note that with the embodiment shown in FIG. 3, a K number ofmanipulators 62 ₁ through 62 _(K) are provided, but these will hereafterbe described as master manipulator 62 unless there is a particular needto distinguish therebetween.

The network 63 enables communication between the slave manipulator 61,the master manipulators 62 ₁ through 62 _(K), and the super doctordatabase 64.

The super doctor database 64 receives the state of action of the mastermanipulator 62 supplied through the network 63, and generatesinformation relating to surgery based on that state. Further, the superdoctor database 64 generates control signals based on the informationrelating to surgery and the state of the master manipulator 62, andcontrols the slave manipulator 61 by supplying this to the slavemanipulator 61 through the network 63.

In the embodiment shown in FIG. 3, the master manipulator 62 correspondsto the operating unit 1003, actuator 1005, and sensor 1007 shown in FIG.1 (and FIG. 2), the network 63 corresponds to the network 1001 in FIG.1, and the super doctor database 64 corresponds to thesynthesizing/integrating unit 1002, driving control unit 1004, andsignal processing unit 1006.

Next, FIG. 4 illustrates an external configuration example of the slavemanipulator 61 and the master manipulator 62 shown in FIG. 3.

That is to say, FIG. 4A illustrates an external configuration example ofthe slave manipulator 61, and FIG. 4B illustrates an externalconfiguration example of the master manipulator 62, respectively.

With the slave manipulator 61 shown in FIG. 4A, a surgery table 1 has arectangular shape, and a patient to be operated upon (a dog in theexample in FIG. 4) is laid thereupon. Also, attached to the surgerytable 1 are a slave manipulator unit 3L and a slave manipulator unit 3R.

The slave manipulator unit 3L and slave manipulator unit 3R (hereaftercollectively referred to as slave manipulator unit 3 unless there is aparticular need to distinguish therebetween) are remotely operated by alater-described master manipulator unit 8L and master manipulator unit8R respectively, so as to perform endermic surgery on the patient laidon the surgery table 1.

The slave manipulator unit 3L is disposed on the left side of thesurgery table 1 (to the left side viewing the surgery table 1 fromabove), and attached to the tip thereof is a tip unit 4L comprisingtreatment equipment such as gripping forceps, scalpel, suturing device,syringe, or the like.

The slave manipulator unit 3R is disposed on the right side of thesurgery table 1, and attached to the tip thereof is a tip unit 4Rcomprising treatment equipment such as gripping forceps, scalpel,suturing device, syringe, or the like.

The slave manipulator unit 3 has a joint structure made up of multiplearms. Further, actuators 8 not shown in FIG. 4) are disposed at eachjoint portion of the slave manipulator 3, so that the arms move with apredetermined degree of freedom by the actuators acting, whereby theslave manipulator unit 3 is capable of assuming various attitudes.

Also, sensors are attached at the joint portions and necessary armportions of the slave manipulator 3 to detect externally-applied forceor torque (including magnitude and direction in either case) on the tipportions 4L and 4R and other portions, and further detecting theattitude of the slave manipulator unit 3 (not shown in FIG. 4).

Note that the sensors and actuators attached to the slave manipulator 3are illustrated in the latter-described FIG. 7 as sensor 75 and actuator76.

Further attached to the surgery table 1 is a camera unit 5 having a CCD(Charge Coupled Device) camera 6 attached to the tip thereof, for takingimages of the inside of the abdominal cavity of the and so forth. Also,the surgery table 1 has attached to the four corners thereof microphones2 ₁ through 2 ₄, for collecting sound.

Now, while FIG. 4A only shown one camera unit 5 in order to avid thedrawing becoming complicated, there are multiple camera units 5 attachedso as to enable observation of the conditions of the patient, the slavemanipulator unit 3, and the like, from various directions.

Also, the camera unit 5 has a joint structure made up of multiple arms,further comprising unshown actuators provided at each of the jointportions. The arm moves with a predetermined degree of freedom due tothe actuators acting, whereby the CCD camera 6 attached to the cameraunit 5 can take pictures (perform imaging) from various viewpoints.

On the other hand, with the master manipulator 62 shown in FIG. 4B,disposed on an operating base 7 are master manipulator units 8L and 8R(hereafter collectively referred to as master manipulator unit 8 whenappropriate) which are operated by the user U.

The master manipulator unit 8L is disposed to the left side of theoperating base 7 (the left side with back to the operating base 7), withan operating unit 9L, which is held by the left hand of the user U andoperated, attached to the tip thereof.

The master manipulator unit 8R is disposed to the right side of theoperating base 7, with an operating unit 9R, which is held by the righthand of the user U and operated, attached to the tip thereof.

Now, the operating units 9L and 9R (hereafter collectively referred toas operating unit 9 as appropriate) correspond to the operating unit1003 in FIG. 1.

The master manipulator unit 8 has a joint structure made up of multiplearms in the same way as the slave manipulator 3, so as to move with apredetermined degree of freedom. Accordingly, the operating unit 9L ofthe master manipulator 8L moves three-dimensionally due to operations bythe left hand of the user U, and the operating unit 9R of the mastermanipulator 8R also moves three-dimensionally due to operations by theright hand of the user U.

Note that actuators (not shown in FIG. 4) are disposed at each jointportion of the master manipulator unit 8, so that the arm moves with apredetermined degree of freedom by the actuators acting, whereby themaster manipulator unit 8 provides predetermined reactive force orpressing force to the user U. Here, the actuators provided on the jointportions correspond to the actuator 1005 shown in FIG. 1.

Also, sensors (not shown in FIG. 4) are provided to the joint portionsand necessary arm portions of the master manipulator unit 8, fordetecting (sensing) force or torque externally applied to the operatingunits 9L and 9R and other portions, and further for detecting theattitude of the master manipulator unit 8. Here, the sensors correspondto the sensor 1007 shown in FIG. 1.

Note that the sensors and actuators attached to the master manipulator62 are illustrated in the later-described FIG. 8 as sensor 85 andactuator 86.

A monitor 10 is provided at the front side of the operating base 7 (thefront direction of the user U in the event that the user U stands infront of the operating base 7 in order to operate the master manipulatorunit 8), so that the user U can view an image displayed thereupon whileoperating the master manipulator unit 8. The monitor 10 displays imagestaken by the CCD camera 6 of the camera unit 5.

A speaker 11 is provided near the monitor 10, so that sound collected bythe microphones 2 ₁ through 2 ₄ attached to the surgery table 1 of theslave manipulator 61 are output from this speaker 11.

Moreover, an operating panel 12 is provided near the monitor 10. Theoperating panel 12 has various types of switches and buttons and thelike attached thereto, which are operated by the user as necessary.Examples of switches or the like on the operating panel 12 include anelectric power switch, a switch for switching the image displayed on themonitor 10, and so forth.

The user U operating the manipulator 92 configured as described abovestands between the operating base 7 and the monitor 10 with his/her backto the operating base 7, and three-dimensionally moves the operatingunit 9L of the master manipulator unit 8L with his/her left hand, andalso three-dimensionally moves the operating unit 9R of the mastermanipulator unit 8R with his/her right hand, while viewing the tip 4 ofthe slave manipulator unit 3 shown on the monitor 10.

In this case, operating signals corresponding to operations of themaster manipulator unit 8 by the user U are supplied to the super doctordatabase 64 via the network 63, and the super doctor database 64processes the operating signals to generate control signals forcontrolling the slave manipulator 61. The control signals are suppliedfrom the super doctor database 64 to the slave manipulator 61 to via thenetwork 63. At the slave manipulator 61, the slave manipulator unit 3acts according to the control signals from the super doctor database 64.Accordingly, the tip 4L of the slave manipulator 3L moves synchronouslywith the movement of the operating unit 6L of the master manipulator 8L,and the tip 4R of the slave manipulator 3R moves synchronously with themovement of the operating unit 6R of the master manipulator 8R, therebyperforming endermic surgery on the patient.

Now, there may be cases wherein there is offset between the actualoperations and the intended operations, even in the event that a userU_(k) attempts to operate the operating unit 9 of the master manipulatorunit 8 so as to pass through a position of action of the operating unit9, determined by the coordinates (X, Y, Z), in a state of rollingrotations, pitching rotations, and yawing rotations, each ofpredetermined angles, as a predetermined state.

Note that yawing rotation is rotation of the X axis and Y axis on the Zaxis as shown in FIG. 5, and the rotation angle thereof is indicated inthe drawing by θ_(y). Pitch rotation is rotation of the Z axis and the Xaxis which has been subjected to yawing rotation by θ_(y) (indicated byX′ in the drawing), on the Y axis which has been subjected to yawing(indicated by Y′ in the drawing), and the rotation angle thereof isindicated in the drawing by θ_(p). Rolling rotation is rotation of the Yaxis which has been subjected to yawing rotation by θ_(y) (indicated byY′ in the drawing) and the Z axis which has been subjected to pitchrotation by θ_(p) (indicated by Z′ in the drawing), on the X axis whichhas been subjected to yawing rotation by θ_(y) and then pitch rotationby θ_(p) (indicated by the X axis in the drawing), and the rotationangle thereof is indicated in the drawing by θ_(r).

For example, in the event that the tip 4 of the slave manipulator unit 3is to be moved from a predetermined position A to a position B over apath (target path) with a certain degree of curvature as shown in FIG.6A, each of the users U₁ through U_(K) attempt to move the operatingunit 9 of the master manipulator unit 8 so as to trace a pathcorresponding to the target path (while there is difference inconfiguration according to the size of the slave manipulator unit 3 andthe master manipulator unit 8, handled here as the same path for thesake of simplification). However, in reality, the operating unit 9 manybe moved so as to trace a path (operating path) off from the target pathindicated by the dotted line in FIG. 6B, as indicated by the solid linein FIG. 6B, although depending on the operating skill level of the usersU₁ through U_(K) with regard to the master manipulator unit 8 (operatingunit 9).

Now, since offset between the operating path and the target pathnormally occurs randomly (since this does not occur with a certaintendency), a path closer to the target path can be obtained by averagingthe multiple operating paths (the operating paths of each of the usersU₁ through U_(K) (FIG. 6B)).

Accordingly, the super doctor database unit 64 synthesizes the contentsof operations at the master manipulators 62 ₁ through 62 _(K) (e.g.,averages), and the slave manipulator 61 (slave manipulator unit 3) isremotely operated based on the synthesized (averaged) operatingcontents. Accordingly, with the slave manipulator 61, the tip 4 of theslave manipulator 3 can be moved to trace a path closer to the targetpath as shown in FIG. 6C.

Note that in FIG. 6A, the dotted line arrow indicates the direction ofthe path, the dotted line arrow in FIG. 6B indicates the direction ofmovement of the operating unit 9, and the dotted line arrow in FIG. 6Cindicates the direction of movement of the tip 4.

Also, the super doctor database unit 64 stores surgery skill informationwhich is skill-related information of past surgery performed by renownedsurgeons from all over the world, classified by symptoms, for example,and the operating contents at the master manipulators 62 ₁ through 62_(K) are corrected based on the surgery skill information, that is tosay, the super doctor database unit 64 controls the slave manipulator 61based on the stored surgery skill information as well as the contents ofoperation of the master manipulators 62 ₁ through 62 _(K).

Accordingly, with a SHARN surgery system, surgery of the highest levelcan be realized by users U₁ through U_(K) which are veterinariansscattered throughout the world performing joint surgery by eachoperating the master manipulators 62 ₁ through 62 _(K), while makingreference to surgery skill information.

Next, FIG. 7 illustrates an example of an electrical configuration ofthe slave manipulator 61 shown in FIG. 3.

A slave manipulator control device 51 is connected to the network 63,and controls the slave manipulator unit 3.

That is to say, the slave manipulator control device 51 is made up of acontrol unit 71, an attitude transition mechanism unit 72, a controlmechanism unit 73, and a communication unit 74.

The control unit 71 takes attitude parameters serving as control signalssent out onto the network 63 by the super doctor database unit 64 andreceived by the control unit 71 through the communication unit 74,supplies these to the attitude transition mechanism unit 72 and alsocalculates the attitude parameters at the tip 4 of the slave manipulatorunit 3 based on the output of the sensor 75 of the slave manipulatorunit 3 supplied via the control mechanism unit 73 and sends these out onthe network 63 via the communication unit 74.

Now, the attitude parameters of the tip 4 of the slave manipulator unit3 are made up of, for example, the X-axial value, Y-axial value, andZ-axial value, for determining the position of the tip 4 (acting point),and the yaw angle θ_(y), pitch angle θ_(p), and roll angle θ_(r), whichdetermine the angle of the tip 4, and so forth. The position of the tip4 is the coordinates on the XYZ orthogonal coordinates system definedwith a predetermined position in space where the tip 4 can move as thepoint of origin, and also, the angle of the tip 4 is defined shown inFIG. 5, with reference to the XYZ orthogonal coordinates system definedregarding the top 4, for example. Note that hereafter, both the positionand angle together will be referred to as attitude, as appropriate.

The control unit 71 calculates the force F1 and torque T1 which the tip4 of the slave manipulator unit 3 has externally received, based on theoutput of the sensor 75 of the slave manipulator unit 3, supplied viathe control mechanism unit 73, and sends this out onto the network 63via the communication unit 74.

The attitude transition mechanism unit 72 generates attitude transitioninformation whereby the attitude of the tip 4 of the slave manipulatorunit 3 makes transition from the current attitude to an attitudecorresponding to the attitude parameters supplied from the control unit71, and sends this to the control mechanism unit 73.

The control mechanism unit 73 generates control signals according to theattitude transition information from the attitude transition mechanismunit 72, and supplies this to the slave manipulator unit 3. The actuator76 of the slave manipulator unit 3 is driven according to the controlsignals supplied from the control mechanism unit 73, and accordingly,the tip 4 assumes an attitude corresponding to the attitude parameterstransmitted from the super doctor database unit 64.

The control mechanism unit 73 also acquires output of the sensor 75 ofthe slave manipulator unit 3, and supplies this to the control unit 71.

The communication unit 74 performs communication control via the network63, and accordingly receives information transmitted from the network 63and supplies this to the control unit 71, and also transmits informationsupplied from the control unit 71 to the network 63.

The input/output control device 52 is connected to the network 63, andtransmits the audio data collected with each of the microphones 2 ₁through 2 ₄, and the image data taken with the cameras 6 ₁ through 6 ₅,via the network 63.

Now, the cameras 6 ₁ through 6 ₅ correspond to the camera 6 in FIG. 4.Also, of the cameras 6 ₁ through 6 ₅, the cameras 6 ₁ through 6 ₄ aresmall-sized CCD cameras which take the affected portion or the like ofthe patient from predetermined viewpoints, and the camera 6 ₅ is anall-directional camera which is capable of taking pictures of the entireperiphery of the slave manipulator unit 4. Now, details of theall-direction camera will be described later.

Next, FIG. 8 illustrates an example of the electrical configuration ofthe master manipulator 62 shown in FIG. 3.

The master manipulator control device 53 is connected to the network 63,and controls the master manipulator unit 8.

That is to say, the master manipulator control device 53 compromises acontrol unit 81, attitude transition mechanism unit 82, controlmechanism unit 83, and communication unit 84.

The control unit 81 calculates the later-described force F0 and torqueT0, based on the later-described reference attitude parameters obtainedvia the communication unit 84, and the force F1 and torque T1 (the forceand torque externally applied to the tip 4 of the slave manipulator unit3), and supplies this to the control mechanism unit 83.

Further, the control unit 81 calculates the attitude parameters of theoperating unit 9 of the master manipulator unit 8, based on the outputof the sensor 85 of the master manipulator unit 8 supplied via thecontrol mechanism unit 83, and transmits these to the network 63 via thecommunication unit 84.

Now, the attitude parameters of the operating unit 9 of the mastermanipulator unit 8 are configured of coordinates made up of X-axialvalues, Y-axial values, and Z-axial values, determining the position ofthe operating unit 9 (acting point), the yaw angle θ_(y), pitch θ_(p),and roll θ_(r), determining the angle of the operating unit 9, and soforth, as with the attitude parameters of the tip 4 of the slavemanipulator 3. The position of the operating unit 9 is coordinates onthe XYZ orthogonal coordinates system defined with a certain position inspace where the operating unit 9 can move as the point of origin, andalso the angle of the tip 4 is defined as shown in FIG. 5 with referenceto the XYZ orthogonal coordinates system defined regarding the operatingunit 9, for example.

Also, the attitude parameters of the operating unit 9 of the mastermanipulator unit 8, and the attitude parameters of the tip 4 of theslave manipulator unit 3 described above, may be defined on the same XYZorthogonal coordinates system, or may be defined on a different XYZorthogonal coordinates system. Note however, that in the event ofdefining the attitude parameters of the tip 4 and the operating unit 9on different XYZ orthogonal coordinates systems, the super doctordatabase unit 64 needs to recognize the correlation between thediffering XYZ orthogonal coordinates systems.

The control mechanism unit 83 generates control signals following theforce F0 and torque T0 supplied from the control unit 81, and sends thisto the master manipulator 8. The actuator 86 of the master manipulator 8is driven according to these control signals, and accordingly, theoperating unit 9 of the master manipulator 8 provides the force F0 andtorque T0 calculated by the control unit 81 to the user U operating theoperating unit 9.

The control mechanism unit 83 also supplies the output of the sensor 85of the master manipulator 8 to the control unit 81.

The communication unit 84 performs communication control via the network63, receives information transmitted from the network 63, and suppliesthis to the control unit 81, and also transmits information suppliedfrom the control unit 81 to the network 63.

The input/output control device 54 is connected to the network 63, andreceives image data and audio data transmitted from the network 63 andsupplies each to the monitor 10 and speaker 11. Accordingly, the imagestaken with the cameras 6 ₁ through 6 ₅ of the slave manipulator 61 aredisplayed on the monitor 10, and the audio collected with themicrophones 2 ₁ through 2 ₄ of the slave manipulator 61 are output fromthe speaker 11.

Also, the input/output control device 54 receives operating signalswhich the operating panel 12 outputs under operations of the user U, andswitches the image to be displayed on the monitor 10 or audio to beoutput from the speaker 11, according to the operating signals. Further,the input/output control device 54 transmits the operating signals fromthe operating panel 12 to the network 63 as necessary.

Next, FIG. 9 illustrates a configuration example of the super doctordatabase unit 64 shown in FIG. 3.

An input/output control unit 21 is connected to the network 63, andperforms communication control via the network 63 such that informationtransmitted from the network 63 is received and supplied to an attitudeparameter registration unit 22 and attitude parameter control unit 28,and information supplied from the attitude parameter control unit 28 istransmitted to the network 63.

Now, information which the input/output control unit 21 receives fromthe network 63 includes, for example, attitude parameters of the tip 4which the slave manipulator 61 transmits, attitude parameters of theoperating unit 9 which the master manipulator 62 transmits, and soforth.

The attitude parameter registration unit 22 recognizes what sort ofsurgical contents operations which the master manipulator 62 attemptedbased on the attitude parameters of the operating unit 9 sent from themaster manipulator 62, supplied from the input/output control unit 21,and registers the recognized surgical contents and the attitudeparameters as surgery information in the surgery information database25.

A surgery development information database 23 stores surgery developmentinformation which is information relating to patients following surgery,regarding surgery performed in nations all over the world includingsurgery with SHARN surgery systems, for example, classified by surgery,for example.

A surgery development information registration unit 24 registers thesurgery development information, by surgery, in the surgery developmentinformation database 23. Note that the surgery development informationregistration unit 24 collects, for example, surgery developmentinformation from databases in hospitals all over the world (includingSHARN surgery systems) storing the contents of medical records and thelike via the network 63, and registers this in the surgery developmentdatabase 23. Also, the surgery development information registration unit24 can also receive direct information of surgery developmentinformation by a predetermined operator performing operations, and inthis case, the surgery development information registration unit 24registers the directly-input surgery development information to thesurgery development database 23.

A surgery information database 25 stores surgery information suppliedfrom the attitude parameter registration unit 22.

An updating unit 26 determines whether or not surgery of the surgerycontents indicated by the surgery information stored in the surgeryinformation database 25 has succeeded, with reference to surgerydevelopment information of the surgery development database 23 and thesurgery skill information database 27 is updated based on the surgeryinformation for surgery which has been successful.

The surgery skill information database 27 stores surgery skillinformation which is information relating to surgical skill suppliedfrom the updating unit 26 and a surgery skill information registeringunit 30. That is, the surgery skill information database 27 storesinformation regarding each surgery, classified by symptoms, on theoptimal way of operating the operating unit 9 for that surgery,operational contents to be restricted for operation of the operatingunit 9, and so forth, as surgery skill information.

The attitude parameter control unit 28 receives attitude parameters ofthe operating unit 9 of the master manipulator 62 received by theinput/output control device 21, corrects the received attitudeparameters while making reference to the surgery sill information via asearch unit 29 as necessary, and supplies these to the input/outputcontrol device 21 as reference attitude parameters.

The search unit 29 searches the surgery skill information stored in thesurgery skill information database 27 under the control of the attitudeparameter control unit 28, and supplies the searched results to theattitude parameter control unit 28.

The surgery skill information registering unit 30 is operated by apredetermined operator for example, and the surgery skill informationcorresponding to the operations thereof is registered in the surgeryskill information database 27.

Next, the processing of the slave manipulator 61 shown in FIG. 7 will bedescribed with reference to the flowchart in FIG. 10.

In step S1, the input/output control device 52 makes the microphones 2 ₁through 2 ₄ and the cameras 6 ₁ through 6 ₅ operational, and startstransmission of the audio data collected with each of the microphones 2₁ through 2 ₄, and the image data taken with each of the cameras 6 ₁through 6 ₅.

That is to say, the input/output control device 52 receives the audiodata collected with each of the microphones 2 ₁ through 2 ₄, and theimage data taken with each of the cameras 6 ₁ through 6 ₅, and transmitsthis to the super doctor database unit 64 via the network 63. Note thattransmission of the image data and audio data continues until anoperation is made at the operating panel 12 of the master manipulator(FIG. 8) to stop the transmission thereof, for example.

Further, in step S1, the slave manipulator control device 51 transmitsthe attitude parameters of the tip 4 of the slave manipulator unit 3 tothe super doctor database unit 64.

That is to say, with the slave manipulator control device 51, thecontrol mechanism unit 73 makes the sensor 75 of the slave manipulatorunit 3 operational, receives the output thereof, and supplies this tothe control unit 71. The control unit 71 obtains the attitude parametersof the tip 4 of the slave manipulator unit 3 based on the output of thesensor 75 supplied from the control mechanism unit 73, and supplies thisto the communication unit 74. The communication unit 74 transmits theattitude parameters from the control unit 71 to the super doctordatabase unit 64 via the network 63.

Subsequently, as described later, the attitude parameters aretransmitted from the super doctor database unit 64 via the network 63,so the communication unit 74 of the slave manipulator control device 51awaits transmission of the attitude parameters from the super doctordatabase unit 64, the flow proceeds to step S2, and receives theattitude parameters.

The attitude parameters are supplied from the communication unit 74 tothe control unit 71, and the control unit 71 controls the slavemanipulator unit 3 in step S3 such that the tip 4 of the slavemanipulator unit 3 assumes the attitude corresponding to the attitudeparameters thereof.

Specifically, the control unit 71 of the slave manipulator controldevice 51 supplies the attitude parameters from the communication unit74 to the attitude transition mechanism 72. The attitude transitionmechanism 72 generates attitude transition information whereby theattitude of the tip 4 of the slave manipulator unit 3 makes transitionfrom the current attitude to an attitude corresponding to the attitudeparameters supplied from the control unit 71, and supplies these to thecontrol mechanism unit 73.

The control mechanism unit 73 generates control signals following theattitude transition information from the attitude transition mechanism72, and sends this to the slave manipulator unit 3. At the slavemanipulator unit 3, the actuator 76 is driven according to the controlsignals from the control mechanism unit 73, and accordingly, theattitude of the tip 4 changes, so as to assume an attitude correspondingto the attitude parameters received with the communication unit 74,i.e., the attitude parameters transmitted from the super doctor database64.

Next, the flow proceeds to step S4, where the slave manipulator controldevice 51 transmits the force F1 and torque T1 applied to the tip 4externally (from the abdominal cavity of the patient) by the attitude ofthe tip 4 of the slave manipulator unit 3 being changed under thecontrol in step S3, transmits these to the super doctor database unit 64via the network 63 along with the attitude parameters of the tip 4.

That is to say, with the slave manipulator control device 51, thecontrol mechanism unit 73 supplies the output of the sensor 75 to thecontrol unit 71, and the control unit 71 calculates the force F1 andtorque T1 externally applied to the tip 4, and the attitude parametersof the tip 4, from the output of the sensor 75. Further, the controlunit 71 supplies the force F1 and the torque T1, and the attitudeparameters, to the communication unit 74, and the communication unit 74transmits the force F1 and the torque T1, and the attitude parameters,to the super doctor database unit 64 via the network 63.

Subsequently, the flow returns to step S2, and the same processing isrepeated.

Accordingly, with the slave manipulator 61, surgery is performed on thepatient by the tip 4 of the slave manipulator unit 3 changing inattitude corresponding to the attitude parameters transmitted from thesuper doctor database 64.

Next, the processing of the master manipulator 62 shown in FIG. 8 willbe described with reference to the flowchart shown in FIG. 11.

As described later, the super doctor database unit 64 transmits theimage data and audio data transmitted from the slave manipulator 61 tothe master manipulator 62 via the network 63, and accordingly in stepS21, reception of the image data and audio data is started.

That is to say, in step S21, the input/output control device 54 receivesthe image data and audio data transmitted from the super doctor databaseunit 64 via the network 63, and supplies the image data to the monitor10 and the audio data to the speaker 11. Accordingly, images taken withthe cameras 6 ₁ through 6 ₅ of the slave manipulator 61 (FIG. 7) isdisplayed on the monitor 10 and audio collected with the microphones 2 ₁through 2 ₄ of the slave manipulator 61 are output from the speaker 11.

Further, as described later, the super doctor database unit 64 transmitsthe attitude parameters of the tip 4 transmitted from the slavemanipulator 61 to the master manipulator 62 via the network 63, so instep S21, the master manipulator control device 53 receives the attitudeparameters, and controls the attitude of the operating unit 9 of themaster manipulator unit 8 based on the attitude parameters.

That is to say, with the master manipulator control device 53, thecommunication unit 84 receives the attitude parameters of the tip 4 ofthe slave manipulator unit 3 transmitted from the super doctor databaseunit 64 via the network 63, and supplies these to the control unit 81.The control unit 81 receives the attitude parameters from thecommunication unit 84, and supplies these to the attitude transitionmechanism unit 82. The attitude transition mechanism unit 82 generatesattitude transition information whereby the attitude of the operatingunit 9 of the master manipulator unit 8 makes transition from thecurrent attitude to an attitude corresponding to the attitude parameterssupplied from the control unit 81, and supplies these to the controlmechanism unit 83. The control mechanism unit 83 generates controlsignals according to the attitude transition information from theattitude transition mechanism unit 82, and sends this to the mastermanipulator unit 8. The actuator 86 of the master manipulator unit 8 isdriven according to the control signals, and accordingly, the attitudeof the operating unit 9 changes so as to assume the same attitude as theattitude of the tip 4 of the slave manipulator unit 3.

Subsequently, in step S22, the attitude parameters of the operating unit9 of the master manipulator unit 8 are transmitted to the super doctordatabase unit 64 via the network 63.

Specifically, with the master manipulator control device 53, the controlmechanism unit 83 receives output of the sensor 85 of the mastermanipulator unit 8, and supplies this to the control unit 81. Thecontrol unit 81 calculates the attitude parameters of the operating unit9 of the master manipulator unit 8 from the output of the sensor 85, andsupplies these to the communication unit 84. The communication unit 84transmits the attitude parameters of the operating unit 9 from thecontrol unit 81 to the super doctor database unit 64 via the network 63.

Now, due to the processing in this step S22, attitude parametersrepresenting the attitude of the operating unit 9 which has been made toassume a certain attitude by the user U operating the operating unit 9of the master manipulator 8 are transmitted to the super doctor databaseunit 64.

The flow then proceeds to step S23, where the communication unit 84 ofthe master manipulator control device 53 receives the attitudeparameters of the tip 4 of the slave manipulator unit 3 transmitted fromthe super doctor database unit 64 via the network as reference attitudeparameters, and the flow proceeds to step S24. In step S24, thecommunication unit 84 of the master manipulator control unit 53 receivesthe force F1 and the torque T1 which the tip 4 has received externally,that is transmitted from the super doctor database unit 64 along withthe attitude parameters of the tip 4. The attitude parameters, and theforce F1 and the torque T1 regarding the tip 4 are supplied from thecommunication unit 84 to the control unit 81, and the flow proceeds fromstep S24 to S25.

In step S25, the control unit 81 calculates the difference between theattitude parameters of the operating unit 9 obtained in step S22 and thereference attitude parameters obtained in step S23, and the flowproceeds to step S26.

In step S26, the control unit 81 calculates a force F2 and a torque T2of a magnitude proportionate to the difference (offset) between theattitude parameters of the operating unit 9 and the reference attitudeparameters calculated in step S25, but in a direction opposite to thedirection of offset, and the flow proceeds to step S27.

In step S27, the control unit 81 adds force and torque proportionate tothe force F1 and the torque T1 obtained in step S24, to each of theforce F2 and torque T2 calculated in step S26, thereby calculating forceF0 and torque T0 to be sensed by the user U operating the operating unit9. That is to say, the control unit 81 calculates the force F0 andtorque T0 according to Expression (1), for example.F ₀ =αF ₁ +F ₂T₀ =βT ₁ +T ₂  (1)

Note that in the event that a greater force is necessary to move theoperating unit 9 of the master manipulator unit 8, for example, ascompared to directly moving the tip 4 of the slave manipulator unit 3(that is to say, when heavy), α and β are values greater than 1, and inthe event that a small force is sufficient for moving the operating unit9 as compared to moving the tip 4 (that is to say, when light), α and βare values smaller than 1. Also, in the event of moving these with thesame force, α and β are 1.

Subsequently, in step S28, the master manipulator unit 8 is controlledsuch that the user U operating the operating unit 9 of the mastermanipulator unit 8 senses the force F0 and torque T0 calculated in stepS27.

That is to say, the control unit 81 supplies the reference attitudeparameters obtained In Step S23 and the force F0 and torque T0 obtainedin step S26 to the attitude transition mechanism unit 82. The attitudetransition mechanism unit 82 generates something which is attitudetransition information for moving the attitude of the operating unit 9of the master manipulator unit 8 so as to make transition from thecurrent attitude to the attitude corresponding to the reference attitudeparameters supplied from the control unit 81, further includinginformation of acceleration (including angular acceleration) whereby theuser can be made to sense the force F0 and torque T0, and supplies thoseto the control mechanism unit 83. The control mechanism unit 83generates control signals following the attitude transition informationfrom the attitude transition mechanism unit 82, and sends this to themaster manipulator unit 8. The actuator 86 of the master manipulatorunit 8 is driven according to these control signals, and accordingly,the operating unit 9 changes the attitude thereof with a predeterminedacceleration, and assumes the same attitude as the tip 4 of the slavemanipulator unit 3 while causing the user U to sense the force F0 andtorque T0.

Subsequently, the flow returns to step S22, and thereafter, the sameprocessing is repeated.

As described above, the force F0 and torque T0 to be sensed by the userU are the added values of the force F2 and torque T2 in the oppositedirection to the offset with a magnitude proportionate to the magnitudeof the offset calculated in step S25, and the force and torqueproportionate to the force F1 and torque T1 which the tip 4 of the slavemanipulator unit 3 has received, thereby enabling the user U to becaused to be conscious of the offset between the attitude parameters ofthe operating unit 9 and the referenced attitude parameters, and thereactive force at the tip 4 of the slave manipulator unit 3.

Accordingly, in the event that the user U operates the operating unit 9of the master manipulator unit 8 so as to assume an attitude differentfrom the attitude corresponding to the reference attitude parameters,the greater the different in attitude is, a great reactive force isreceived, so the user can recognize in a sensory manner how far his/heroperations are from the actual attitude of the tip 4 of the slavemanipulator unit 3.

Or, the user U can be made to sense only the force F2 and torque T2, forexample.

Next, the processing of the super doctor database unit 64 shown in FIG.9 will be described with reference to the flowchart in FIG. 12.

First, in step S41, the input/output control device 21 starts receptionof the image data and audio data from the slave manipulator 61 andtransmission of the image data and audio data to the master manipulator62.

That is, as described in FIG. 10, in step S1, at the slave manipulator61 (FIG. 7), transmission of the image data taken with the cameras 6 ₁through 6 ₅ and audio data collected with the microphones 2 ₁ through 2₄ to the super doctor database unit 64 is started, so the input/outputcontrol device 21 of the super doctor database unit 64 starts receptionof the image data and audio data transmitted from the slave manipulator61 via the network 63.

Further, the input/output control device 21 starts transmission of theimage data and audio data received from the slave manipulator 61 to themaster manipulator 62. Thus, reception of the image data and audio datatransmitted from the input/output control device 21 is started as themaster manipulator 62, in step S21 in FIG. 11 described above.

Also, in step S41, the input/output control device 21 receives theattitude parameters transmitted from the slave manipulator 61 in step S1in FIG. 10 (the attitude parameters of the tip 4 of the slavemanipulator unit 3), and transmits the attitude parameters thereof tothe master manipulator 62 as reference attitude parameters. Now, thereference attitude parameters are received at the master manipulator 62instep S21 in FIG. 11 described above, and accordingly, the operatingunit 9 (the operating unit 9 of the master manipulator unit 8) assumesan attitude corresponding to the reference parameters, i.e., the sameattitude as the attitude of the tip 4 of the slave manipulator unit 3.

Subsequently, the flow proceeds to step S42, where the input/outputcontrol device 21 awaits and receives the attitude parameters regardingthe tip 4 and the force F1 and torque T1 from the slave manipulator 61,transmitted in step S4 in FIG. 10, and transmits the attitude parametersas reference attitude parameters along with the force F1 and torque T1to the master manipulator 62. Now, at the master manipulator 62, theattitude parameters regarding the tip 4 are received in step S23 in FIG.11, and the force F1 and torque T1 regarding the tip 4 are received instep S24 in FIG. 11.

The flow then proceeds to step S43, where the input/output controldevice 21 receives the attitude parameters of the operating unit 9transmitted from each of the master manipulators 62 ₁ through 62 _(K) instep S22 of FIG. 11, and supplies these to the attitude parameterregistration unit 22 and attitude parameter control unit 28, and theflow proceeds to step S44.

In step S44, the attitude parameter registration unit 22 recognizes thesurgery contents which the users U₁ through U_(K) are attempting, basedon the attitude parameters of the respective master manipulators 62 ₁through 62 _(K). Further, the attitude parameter registration unit 22registers the attitude parameters corresponding to the surgery contentsin the surgery information database 25, and the flow proceeds to stepS45.

In step S45, the attitude parameter control unit 28 calculates thescattering of the attitude parameters of the operating unit 9 from eachof the master manipulators 62 ₁ through 62 _(K), and determines whetheror not the scattering is a predetermined threshold value or higher.

In step S45, in the event that determination is made that the scatteringof the attitude parameters of the operating unit 9 from each of themaster manipulators 62 ₁ through 62 _(K) do not reach or exceed thepredetermined threshold value, i.e., in the event that the users U₁through U_(K) of each of the master manipulators 62 ₁ through 62 _(K)are each making similar operations, the flow proceeds to step S46, wherethe attitude parameter control unit 28 obtains a weighting additionvalue such as an averaged value for example, for the attitude parametersof each of the master manipulators 62 ₁ through 62 _(K), and the flowproceeds to step S47.

Now, weighting addition values for the multiple attitude parameters mayinvolve calculating an average value wherein the weighting for each ofthe multiple attitude parameters are the same, or may be calculated withdifference weighting values for each of the multiple attitudeparameters. That is to say, the weighting of the attitude parameters maybe weighted according to the degree of experience of the user U_(k)operating the master manipulator 62 _(k) regarding which attitudeparameters have been transmitted, for example. Note that weighting basedon experience or the like may be set by operating the operating panel 12of the master manipulator panel 62 (FIG. 8), for example.

In step S47, the attitude parameter control unit 28 searches the surgeryskill information database 27 with the search unit 29, so as todetermine whether or not surgery skill information which would limit thesurgery with the attitude parameters having the weighting additionvalues obtained in step S46 is registered in the surgery skillinformation database 27.

In step S47, in the event that determination is made that there is nosurgery skill information registered in the surgery skill informationdatabase 27 which would limit the surgery with attitude parameters whichare weighting addition values, the flow skips step S48 and proceeds tostep S49, where the attitude parameter control unit 28 supplies theattitude parameters which are weighting addition values to theinput/output control device 21, and the input/output control device 21transmits the attitude parameters from the attitude parameter controlunit 28 to the slave manipulator 61. Subsequently, the flow returns tostep S42, and thereafter, the same processing is repeated.

Accordingly, in this case, the movement of the tip 4 of the slavemanipulator 61 follows the averaged movement of the users U₁ throughU_(K) of the master manipulators 62 ₁ through 62 _(K).

Also, in step S47, in the event that judgment is made that surgery skillinformation for restricting the surgery with attitude parameters whichare weighting addition values is registered to the surgery skillinformation database 27, that is, in the event that the contents of thesurgery according to attitude parameters which are weighting additionvalues are not found in past cases, or in the event that acceptablesurgery development was not obtained in the past, the flow proceeds tostep S48, and the attitude parameter control unit 28 corrects theattitude parameters which are weighting addition values according to thesurgery skill information for restricting the surgery by the attitudeparameters thereof, so as to bring into the restricted range of thesurgery skill information. The flow then proceeds to step S49, theattitude parameter control unit 28 transmits the corrected attitudeparameters to the slave manipulator 61 via the input/output controldevice 21, the flow returns to step S42, and the same processing isrepeated thereafter.

Accordingly, in this case, the movement of the tip 4 of the slavemanipulator 61 follows the averaged movement of the users U₁ throughU_(K) of the master manipulators 62 ₁ through 62 _(K) which has beencorrected based on the surgery skill information.

On the other hand, in the event that it is determined in step S45 thatthe scattering of the attitude parameters of the operating unit 9 fromeach of the master manipulators 62 ₁ through 62 _(K) is equal to orexceeds the predetermined threshold value, i.e., in the event that oneor more of the users U₁ through U_(K) of the master manipulators 62 ₁through 62 _(K) make different operations different to an extent thatthe operations cannot be said to be the same as those of other users,the flow proceeds to step S50, where the attitude parameter control unit28 detects the attitude parameters different from the other attitudeparameters (the attitude parameters causing the increased scattering)(hereafter referred to as unique parameters) from the attitudeparameters of each of the operating units 9, from each of the mastermanipulators 62 ₁ through 62 _(K).

Here, in order to simplify description, we will say that there is oneattitude parameter which is a unique parameter.

In the event that a unique parameter is detected in step S50, the flowproceeds to step S51, where the attitude parameter control unit 28controls the updating unit 26 so as to determine whether or not therehas been a case of successful surgery with the surgery contentsrepresented by the surgery information corresponding to the uniqueparameter stored in the surgery information database 25, by makingreference to the surgery development information of the surgerydevelopment information database 23.

In step S51, in the event that determination is made that there has beenno past case wherein surgery has been successful with the surgerycontents represented by the surgery information corresponding to theunique parameter, the flow proceeds to step S52, and concluding that theunique parameter is caused due to an erroneous operation of theoperating unit 9, the attitude parameter control unit 28 obtains aweighted addition value such as an average value for example, from theattitude parameters from the operating unit 9 from each of the mastermanipulators 62 ₁ through 62 _(K) but excluding the unique parameter,and the flow proceeds to step S47.

Subsequently, in steps S47 through S49, processing the same as describedabove is performed with regard to weighted addition values of theattitude parameters from the operating unit 9 from each of the mastermanipulators 62 ₁ through 62 _(K) excluding the unique parameters, andthe flow returns to step S42.

Accordingly, in this case, the movement of the tip 4 of the slavemanipulator 61 follows the averaged movement of the users U₁ throughU_(K) of the master manipulators 62 ₁ through 62 _(K) from whichoperations markedly different from other operations are removed (orwherein the averaged operations are corrected based on the surgery skillinformation).

On the other hand, in step S51, in the event that it is determined thatthere have been past cases wherein surgery following the surgery contentrepresented by the surgery information corresponding to the uniqueparameter has been successful, the flow proceeds to step S53, where theattitude parameter control unit 28 updates the surgery skill informationdatabase 27 by controlling the update unit 26.

That is, in this case, the updating unit 26 generates optimal operatingmethods (procedures) for the operating unit 9 to carry out the surgery,or generates operating contents or the like to be restricted asoperations of the operating unit 9 as surgery skill information, basedon the surgery contents represented by the surgery informationcorresponding to the unique parameter, and supplies and stores this tothe surgery skill information database 27, thereby updating the surgeryskill information database 27.

The flow then proceeds to step S54, where the attitude parameter controlunit 28 controls the input/output control device 21 so as to transmitthe attitude parameter which is a unique parameter to the slavemanipulator 61, following which the flow returns to step S42 and thesame processing is repeated thereafter.

Accordingly, in this case, the movement of the tip 4 of the slavemanipulator 61 follows operations markedly different from otheroperations of the users U₁ through U_(K) of the master manipulators 62 ₁through 62 _(K).

That is to say, in the event that one of the users U₁ through U_(K) ofthe master manipulators 62 ₁ through 62 _(K) makes operations differentfrom other users (here, different to an extent that the operationscannot be said to be the same operations), and moreover there is aprecedent of the surgery succeeding with such an operation, the movementof the tip 4 of the slave manipulator 61 follows the operations of theone user which has performed operations different to those of the otherusers.

Consequently, in the event that one of the users U₁ through U_(K) is arenowned surgeon with highly-experienced surgery skills, as comparedwith the other users, and in the event that the user which is therenowned surgeon makes operations different from those of the other usesbased on the nearest surgery skills, the tip 4 of the slave manipulator61 moves corresponding to the operations of the user which is therenowned surgeon.

Accordingly, with the SHARN surgery system shown in FIG. 3, the surgeryskill information of the surgery skill information database 27 in thesuper doctor database unit 64 is continuously updated to a better one,by the attitude parameters based on the operations of each of the mastermanipulators 62 ₁ through 62 _(K). Further, with the super doctordatabase unit 64, attitude parameters to be provided to the slavemanipulator 61 (reference attitude parameters) are generated from theattitude parameters based on the operations of each of the mastermanipulators 62 ₁ through 62 _(K), and the surgery skill information,and while surgery is performed on the patient at the slave manipulator61 by the attitude of the top 4 being controlled following the referenceattitude parameters, a load is placed on the users U₁ through U_(K)operating the operating unit 9 so as to correct, as if it were, theoffset between the operations of each of the master manipulators 62 ₁through 62 _(K) and the reference attitude parameters.

As a result, surgery of the highest level is performed at the slavemanipulator 61, based on skills from surgeons which are multiple users,and surgery skill information which is built upon past surgery and iscontinuously updated.

Also, at the master manipulator 62, the operations of the veterinarianwhich is a user are corrected so as to match the movement of the tip 4in the actual surgery, thereby improving the surgery skills of theveterinarian which is a user in a short time.

That is to say, with the SHARN surgery system shown in FIG. 3, thesurgery skill information of the super doctor database unit 64 isupdated into a better one based on the operations of the mastermanipulators 62 ₁ through 62 _(K), and further, feedback, as if it were,can be received at the master manipulators 62 ₁ through 62 _(K), so thatthe operations of the users are corrected so as to match the surgeryskill information.

Accordingly, at each of the master manipulators 62 ₁ through 62 _(K),high functionality consisting of the skills of the surgeons which areusers being improved in a short time, which could not be realized inindividual cases, can be said to be realized, due to the mastermanipulators 62 ₁ through 62 _(K), and the super doctor database unit64, surrounded by the dotted line in FIG. 13, acting as if they were asingle apparatus, with the surgery skill information of the super doctordatabase unit 64 being updated into a better one based on the operationsof each of the master manipulators 62 ₁ through 62 _(K), whereby each ofthe master manipulators 62 ₁ through 62 _(K) share the processing ofupdating the surgery skill information into a better one throughcollaboration.

Conversely, in the event that the super doctor database unit 64 onlystores the operation contents of each of the master manipulators 62 ₁through 62 _(K), and gives feedback to the master manipulators 62 ₁through 62 _(K), this is only simple sharing of the processing of themaster manipulators 62 ₁ through 62 _(K) supplying the operations madeby users to the super doctor database unit 64, and accordingly, thiscannot realize the high functionality at each of the master manipulators62 ₁ through 62 _(K) of the skills of the veterinarians as users beingimproved.

Next, FIG. 14 illustrates a configuration example of a SHARN designsystem serving as a design system to which the SHARN system has beenapplied.

This SHARN design system is a manipulator system for designing, and ismade up of a pseudo slave manipulator 41, master manipulators 42 ₁through 42 _(K), a network 43, and a super designer database unit 44.

The pseudo slave manipulator 41 acts based on signals supplied via thenetwork 43, and performs certain designing on a subject of design. Here,a PDA (Personal Digital Assistance) for example, is the subject ofdesign. Note however, that the subject of design is not restricted toPDAs.

The master manipulator 42 _(k) acts according to the operations of theuser U_(k) and information supplied from the super designer databaseunit 44 through the network 43. Also, the master manipulator 42 _(k)detects the state of action thereof, and transmits the detection resultsto the super designer database unit 44 via the network 43.

Note that while in the embodiment shown in FIG. 14, a K number of mastermanipulators 42 ₁ through 42 _(K) are provided, these will be describedas master manipulator 42 hereafter unless there is a particular need todistinguish therebetween.

The network 43 enables communication between each of the pseudo slavemanipulator 41, master manipulators 42 ₁ through 42 _(K), and the superdesigner database unit 44.

The super designer database unit 44 receives the state of actions of themaster manipulator 42 supplied via the network 43, and generatesinformation relating to design based on that state. Further, the superdesigner database unit 44 generates control signals based on theinformation relating to design and the state of the master manipulator42, and supplies these to the pseudo slave manipulator 41 via thenetwork 43, thereby controlling the pseudo slave manipulator 41.

Note that in the embodiments shown in FIG. 14, the master manipulator 42corresponds to the operating unit 1003 shown in FIG. 1 (and FIG. 2), theactuator 1005, and the sensor 1007, the network 43 corresponds to thenetwork 1001 shown in FIG. 1, and the super designer database unit 44corresponds to the synthesizing/integrating unit 1002, driving controlunit 1004, and signal processing unit 1006 shown in FIG. 1.

Also, with the embodiment shown in FIG. 14, the pseudo slave manipulator41 is not a manipulator which performs actual work, but a manipulatorwhich performs work in a virtual manner according to the mastermanipulator 42. Note however, that the pseudo slave manipulator 41 maybe formed as a manipulator which performs actual work.

Next, FIG. 15 illustrates an example of the electrical configuration ofthe pseudo slave manipulator 41 shown in FIG. 14. In the drawing,portions which corresponding to the case of the slave manipulator 61shown in FIG. 7 are denoted with the same reference numerals, anddescription thereof will be omitted hereafter as appropriate.

The pseudo slave manipulator unit 77 has the same functions as those ofthe slave manipulator unit 3 shown in FIG. 7 in a pseudo manner,realized by a CPU executing a program, for example. The pseudo slavemanipulator unit 77 performs various actions in a pseudo manner withregard to the object of design, following the control of the slavemanipulator control unit 51.

That is to say, the pseudo slave manipulator unit 77 performs varioustypes of processing with a virtual tip (equivalent to the tips 4L and 4Rof the slave manipulator 61 in FIG. 4A), with regard to a virtual PDAdetermined from PDA design data stored in a design information database78, following the control of the slave manipulator control unit 41.

Also, the pseudo slave manipulator unit 77 operates on a virtual PDAwith the virtual tip in the same way as the slave manipulator unit 3shown in FIG. 7, whereby signals corresponding to the force F1 andtorque T1 which the virtual tip receives from the virtual PDA, i.e.,signals corresponding to output of the sensor 75 of the slavemanipulator unit 3 shown in FIG. 7, are calculated, and supplied to acontrol mechanism unit 73.

The design information database 78 stores design information which isdata for determining external configuration of the virtual PDA. That isto say, the design information database 78 stores PDA design informationwhich the users U₁ through U_(K) of the master manipulators 42 ₁ through42 _(K) have designed by operating the master manipulators 42 ₁ through42 _(K).

An image generating unit 79 generates image data representing the stateof the virtual tip and the like, based on internal data of the pseudoslave manipulator unit 77. Further, the image generating unit 79generates image data of the virtual PDA from the design informationstored in the design information database 78. The image generating unit79 then synthesizes the image data of the virtual tip@ and the like, andthe image data of the virtual PDA, thereby generating image data whichwould have been taken by the cameras 6 ₁ through 6 ₂ of the slavemanipulator 61 in FIG. 7 had the pseudo slave manipulator unit 77 beenan actual manipulator and the work done on an actual PDA. This imagedata is supplied from the image generating unit 79 to an input/outputcontrol device 80.

The input/output control device 80 is connected to the network 43, andthe image data supplied from the image generating unit 79 is transmittedto the super designer database 44 via the network 43. Also, theinput/output control device 80 transmits design information regardingthe PDA designed by each of the users U₁ through U_(K) that is stored inthe design information database 78, to the super designer database 44via the network 43.

Next, FIG. 16 illustrates an example of the electrical configuration ofthe master manipulator 42 shown in FIG. 14. Note that the portions inthe drawing which correspond to the master manipulator 62 shown in FIG.8 are denoted with the same reference numerals, and description thereofwill be omitted hereafter as appropriate. That is to say, the mastermanipulator 42 shown in FIG. 14 is configured in the same way as themaster manipulator 62 shown in FIG. 8, except that no speaker 11 isprovided. Accordingly, while not shown in the drawing, the externalconfiguration of the master manipulator 42 is basically the same as theexternal configuration of the master manipulator 62 shown in FIG. 4B.

Next, FIG. 17 illustrates a configuration example of the super designerdatabase unit 44 shown in FIG. 14. Note that the portions in the drawingwhich correspond to the super doctor database unit 64 shown in FIG. 9are denoted with the same reference numerals, and description thereofwill be omitted hereafter as appropriate.

A design information registration unit 91 receives design informationregarding the PDA designed by each of the users U₁ through U_(K), whichis stored in the design information database 78 and transmitted from theinput/output control device 80 of the pseudo slave manipulator 41 inFIG. 15 over the network 43, supplies this to a design informationdatabase 92, and registers it. Note that transfer of the designinformation regarding the PDA which the user U_(K) has designed, fromthe pseudo slave manipulator 41 to the super designer database unit 44,is performed by the user U_(K), for example, making operations with theoperating unit 12 of the master manipulator 42 so as to register his/herown PDA design.

The design information database 92 stores design information from thedesign information registration unit 91.

A design evaluating unit 93 evaluates the PDA design determinedaccording to design information stored in the design informationdatabase 92.

That is to say, the design evaluating unit 93 discloses the externalconfiguration of a PDA determined by the design information stored inthe design information database 92, for example, on a web page on theInternet, for example, so as to hold a popularity vote, so that the PDAdesign is evaluated based on the results of the vote. Or, in the eventthat the PDA determined by the design information stored in the designinformation database 92 is already being sold as a product, for example,the PDA design is evaluated by collecting the number of PDA units soldvia the network 43, based on the number of units sold.

A unique design information database 94 stores unique design informationsupplied from the attitude parameter registration unit 22, and the voguedesign information database 95 stores the vogue design informationsupplied from the updating unit 26.

Now, vogue design information means design information stored in thedesign information database 92 with high evaluation made by the designevaluation unit 93, and unique design information means designinformation which is off, as if it were, from the vogue designinformation (hereafter referred to as unique design as appropriate).

Next, FIG. 18 illustrates PDA design information stored in the designinformation database 78 shown in FIG. 15, the design informationdatabase 92 shown in FIG. 17, the unique design information database 94,and the vogue design information database 95.

PDA design information which can be employed include, as shown in FIG.18A, the width-wise length p1 of the PDA, the longitudinal length p2,the width-wise length p3 of the PDA which is the display unit of thePDA, the longitudinal length p4 thereof, the position p6 of the centerof gravity of the PDA, the positions p7 and p8 of buttons provided onthe PDA, and so forth.

Also, as shown in FIG. 18B, in the event that the PDA has an arc-shapedroundness, the radii of the arcs, p9, p10, p11, p12, and so forth, andbe employed as PDA design information.

Next, the processing of the pseudo slave manipulator 41 shown in FIG. 15will be described with reference to the flowchart shown in FIG. 19.

In step S61, the input/output control device 80 starts transmission ofimage data generated at the image generating unit 79.

That is to say, Upon the user U_(k) of the master manipulator 42 _(k)(FIG. 16) so as to start designing, an operating signal corresponding tothe operation thereof (hereafter referred to as start signal) istransmitted from the input/output control device 54 to the pseudo slavemanipulator 41 and super designer database unit 44 via the network 43.

The start signal from the master manipulator 42 _(k) is received by theinput output control device 80 of the pseudo slave manipulator 41, andin this case, the input output control device 80 supplies this to theimage generating unit 79 to generate the image data of the user U_(k).Accordingly, the image generating unit 79 starts generating of imagedata of the user U_(k), and the input output control device 80 startstransmission of image data generated for the user U_(k) (hereafterreferred to as user U_(k) image data where appropriate). Note that thetransmission of the image data is continued until the operating panel 12of the maser manipulator 42 _(k) (FIG. 16) is subsequently operated soas to stop the transmission thereof, for example.

Further, in step S61, the slave manipulator control device 51 transmitsthe attitude parameters at the virtual tip of the pseudo slavemanipulator unit 77 to the super designer database unit 44.

That is to say, at the slave manipulator control device 51, the controlmechanism unit 73 receives signals relating to the state of the virtualtip which the pseudo slave manipulator unit 77 outputs (hereafterreferred to as state signals as appropriate), and supplies these to thecontrol unit 71. The control unit 71 obtains the attitude parameters ofthe virtual tip of the pseudo slave manipulator unit 77 based on thestate signals supplied from the control mechanism unit 73, and suppliesthese to the communication unit 74. The communication unit 74 transmitsthe attitude parameters from the control unit 71 to the super designerdatabase unit 44 via the network 43.

Subsequently, as described later, attitude parameters are transmittedfrom the super designer database unit 44 through the network 43, so thecommunication unit 74 of the slave manipulator control device 51 awaitsthe transmission of the attitude parameters from the super doctordatabase unit 64, the flow proceeds to step S62, and the attitudeparameters are received.

The attitude parameters are supplied from the communication unit 74 tothe control unit 71, and the control unit 71 controls the pseudo slavemanipulator unit 77 in step S63 so that the virtual tip of the pseudoslave manipulator unit 77 assumes an attitude corresponding to theattitude parameters.

Specifically, the control unit 71 of the slave manipulator controldevice 51 supplies the attitude parameters from the communication unit74 to the attitude transition mechanism unit 72. The attitude transitionmechanism unit 72 generates attitude transition information for causingthe attitude of the virtual tip of the pseudo slave manipulator unit 77to make transition from the current attitude to the attitudecorresponding to the attitude parameters supplied from the control unit71, and supplies these to the control mechanism unit 73.

The control mechanism unit 73 generates control signals according to theattitude transition information from the attitude transition mechanismunit 72, and transmits these to the pseudo slave manipulator unit 77.The pseudo slave manipulator unit 77 changes the attitude of the virtualtip according to the control signals from the control mechanism unit 73,so as to assume an attitude corresponding to the attitude parametersreceived by the communication unit 74.

The flow then proceeds to step S64, where the slave manipulator controldevice 51 transmits the force F1 and torque T1 externally applied (fromthe virtual PDA which is the object of design) due to the virtual tip ofthe pseudo slave manipulator unit 77 changing the attitude thereof inthe control in step S63 to the super designer database unit 44 via thenetwork 43, along with the attitude parameters of the virtual tipthereof.

That is to say, at the slave manipulator control device 51, the controlmechanism unit 73 supplies state signals output from the pseudo slavemanipulator unit 77 to the control unit 71, and the control unit 71calculates the force F1 and torque T1 which the virtual tip hasexternally received, and the attitude parameters of the virtual tip,from the state signals. Further, the control unit 71 supplies the forceF1 and torque T1, and the attitude parameters, to the communication unit74, and the communication unit 74 transmits the force F1 and torque T1,and the attitude parameters, to the super designer database unit 44 viathe network 43.

Subsequently, the flow returns to step S62, and the same processing isrepeated hereafter.

Accordingly, at the pseudo slave manipulator 41, the PDA is subjected tobeing design by the attitude of the virtual tip of the pseudo slavemanipulator unit 77 changing according to the attitude parameterstransmitted from the super designer database unit 44.

Note that in step S64, an arrangement may be made wherein the force F1and the torque T1 are calculated assuming that the PDA is formed ofplastic foam or ABS (Acrylonitrile Butadiene Styrene) resin or the likeused as material for models, for example.

Next, the processing of the master manipulator 42 shown in FIG. 16 willbe described with reference to the flowchart shown in FIG. 20.

As described later, the super designer database unit 44 transmits imagedata transmitted from the pseudo slave manipulator 41 to the mastermanipulator 42 via the network 43, so in step S71, reception of theimage data is started.

That is to say, in step S71, the input/output control device 54 receivesimage data transmitted from the super designer database unit 44 via thenetwork 43, and supplies the image data to the monitor 10. Accordingly,the monitor 10 displays an image illustrating the way in which thepseudo slave manipulator unit 77 (FIG. 15) performs processing on thePDA.

Further, as describe later, the super designer database unit 44transmits the attitude parameters of the virtual tip, transmitted fromthe pseudo slave manipulator 41, to the master manipulator 42 via thenetwork 43, so in step S71, the master manipulator control device 53receives the attitude parameters, and controls the attitude of theoperating unit 9 of the master manipulator unit 8 based on the attitudeparameters.

That is, at the master manipulator control device 53, the communicationunit 84 receives the attitude parameters of the virtual tip of thepseudo slave manipulator unit 77 transmitted from the super designerdatabase unit 44 via the network 43, and supplies these to the controlunit 81. The control unit 81 receives the attitude parameters from thecommunication unit 84, and supplies these to the attitude transitionmechanism unit 82. The attitude transition mechanism unit 82 generatesattitude transition information for causing the attitude of theoperating unit 9 of the master manipulator unit 8 to make transitionfrom the current attitude to the attitude corresponding to the attitudeparameters supplied from the control unit 81, and supplies this to thecontrol mechanism unit 83. The control mechanism unit 83 generatescontrol signals according to the attitude transition information fromthe attitude transition mechanism unit 82, and sends these to the mastermanipulator unit 82. The actuator 86 of the master manipulator unit 8 isdriven according to these control signals, and accordingly, theoperating unit 9 changes in attitude so as to assume the same attitudeas the attitude of the virtual tip of the pseudo slave manipulator unit77.

Subsequently, in step S72, the attitude parameters of the operating unit9 of the master manipulator unit 8 are transmitted to the super designerdatabase unit 44 via the network 43.

Specifically, at the master manipulator control device 53, the controlmechanism unit 83 receives the output of the sensor 85 of the mastermanipulator unit 8, and supplies this to the control unit 81. Thecontrol unit 81 calculates the attitude parameters of the operating unit9 of the master manipulator unit 8 from the output of the sensor 85, andsupplies these to the communication unit 84. The communication unit 84transmits the attitude parameters of the operating unit 9 from thecontrol unit 81 to the super designer database unit 44 via the network43.

Due to the processing in this step S72, the attitude parametersrepresenting the attitude of the operating unit 9 of the mastermanipulator unit 8 changed owing to user operations are transmitted tothe super designer database unit 44.

The flow then proceeds to step S73, where the communication unit 84 ofthe master manipulator control device 53 receives the attitudeparameters of the virtual tip of the pseudo slave manipulator unit 77transmitted from the super designer database unit 44 via the network 43,as reference attitude parameters, and then the flow proceeds to stepS74. In step S74, the communication unit 84 of the master manipulatorcontrol device 53 receives the force F1 and the torque T1 which thevirtual tip has externally received, transmitted from the super designerdatabase unit 44 along with the attitude parameters of the virtual tip.The attitude parameters regarding the virtual tip, and the force F1 andtorque T1, are supplied from the communication unit 84 to the controlunit 81, and the flow proceeds from step S74 to S75.

In step S75, the control unit 81 calculates the difference between theattitude parameters of the operating unit 9 obtained in step S72 and thereference attitude parameters obtained in step S73, and the flowproceeds to step S76.

In step S76, the control unit 81 calculates the force F2 and torque T2of a magnitude proportionate to the difference (offset) between theattitude parameters of the operating unit 9 and the reference attitudeparameters calculated in step S75, but in a direction opposite to thedirection of offset, and the flow proceeds to step S77.

In step S77, the control unit 81 adds force and torque proportionate tothe force F1 and the torque T1 obtained in step S74, to each of theforce F2 and torque T2 calculated in step S76, thereby calculating forceF0 and torque T0 to be sensed by the user U operating the operating unit9. That is to say, the control unit 81 calculates the force T0 andtorque T0 according to the above-described Expression (1), for example.

Subsequently, in step S78, the master manipulator unit 8 is controlledsuch that the user U_(k) operating the operating unit 9 of the mastermanipulator unit 8 senses the force F0 and torque T0 calculated in stepS77.

That is to say, the control unit 81 supplies the reference attitudeparameters obtained In step S73 and the force F0 and torque T0 obtainedin step S76 to the attitude transition mechanism unit 82. The attitudetransition mechanism unit 82 generates something which is attitudetransition information for moving the attitude of the operating unit 9of the master manipulator unit 8 so as to make transition from thecurrent attitude to the attitude corresponding to the reference attitudeparameters supplied from the control unit 81, further includinginformation of acceleration (including angular acceleration as describedabove) whereby the user can be made to sense the force F0 and torque T0,and supplies this to the control mechanism unit 83. The controlmechanism unit 83 generates control signals following the attitudetransition information from the attitude transition mechanism unit 82,and sends this to the master manipulator unit 8. The actuator 86 of themaster manipulator unit 8 is driven according to these control signals,and accordingly, the operating unit 9 changes the attitude thereof witha predetermined acceleration, and assumes the same attitude as thevirtual tip of the pseudo slave manipulator unit 77 while causing theuser U to sense the force F0 and torque T0.

Subsequently, the flow returns to step S72, and thereafter, the sameprocessing is repeated.

As described above, the force F0 and torque T0 to be sensed by the userU are the added values of the force F2 and torque T2 in the oppositedirection to the offset with a magnitude proportionate to the magnitudeof the offset calculated in step S75, and the force and torqueproportionate to the force F1 and torque T1 which the virtual tip of thepseudo slave manipulator unit 77 has received, thereby enabling the userU to be caused to be conscious of the offset between the attitudeparameters of the operating unit 9 and the referenced attitudeparameters, and the reactive force at the virtual tip of the pseudoslave manipulator unit 77.

Accordingly, in the event that the user U_(k) operates the operatingunit 9 of the master manipulator unit 8 so as to assume an attitudedifferent from the attitude corresponding to the reference attitudeparameters, the greater the difference in attitude is, a great reactiveforce is received, so the user can recognize in a sensory manner how farhis/her operations are from the actual attitude of the virtual tip ofthe pseudo slave manipulator unit 77.

Next, the processing of the super designer database unit 44 shown inFIG. 17 will be described with reference to the flowchart in FIG. 21.Note that the super designer database unit 44 performs processing foreach master manipulator 42 _(k) transmitting start signals thereto, soas to control the master manipulator 42 k and the pseudo slavemanipulator 41, according to the flowchart shown in FIG. 17 (hereafterreferred to as control processing, as appropriate). Accordingly, theabove processing is performed for each master manipulator 42 _(k) whichhas transmitted a start signal, at the pseudo slave manipulator 41 aswell.

First, in step S91, the input/output control device 21 starts receptionof image data from the pseudo slave manipulator 41 and transmission ofthat image data to the master manipulator 42 _(k).

That is, as described with FIG. 19, in step S61, the pseudo slavemanipulator 41 (FIG. 15) starts transmission of image data to the superdesigner database unit 44, so the input/output control device 21 of thesuper designer database unit 44 starts reception of image data beingtransmitted thus from the pseudo slave manipulator 41 via the network43.

Further, the input/output control device 21 starts transmission of theimage data received from the pseudo slave manipulator 41 to the masermanipulator 42 _(k). Now, at the master manipulator 42, reception of theimage data transmitted from the input/output control device in this wayis started in step S71 in FIG. 20 described above.

Also, in step S91, the input/output control device 21 receives theattitude parameters transmitted from the pseudo slave manipulator 41 instep S61 in FIG. 19 (the attitude parameters of the virtual tip of thepseudo slave manipulator 77), and transmits the attitude parameters asreference attitude parameters to the master manipulator 42 _(k). Here,the reference attitude parameters are received by the master manipulator42 _(k) in step S71 in FIG. 20 above, whereby the operating unit 9 (theoperating unit 9 of the maser manipulator unit 8) assumes an attitudecorresponding to the reference attitude parameters, i.e., the sameattitude as the virtual tip of the pseudo slave manipulator unit 77.

Subsequently, the flow proceeds to step S92, where the input/outputcontrol device 21 awaits and receives the attitude parameters regardingthe virtual tip and the force F1 and torque T1 transmitted from thepseudo slave manipulator 41 (the attitude parameters of the virtual tipof the pseudo slave manipulator unit 77) in step S64 in FIG. 19, andtransmits the attitude parameters as reference attitude parameters alongwith the force F1 and torque T1 to the master manipulator 42 _(k). Now,at the master manipulator 42 _(k), the attitude parameters regarding thevirtual tip are received in step S73 in FIG. 20, and the force F1 andtorque T1 regarding the virtual tip are received in step S74 in FIG. 20.

The flow then proceeds to step S93, where the input/output controldevice 21 receives the attitude parameters of the operating unit 9transmitted from the master manipulator 42 _(k) in step S72 of FIG. 20,and supplies these to the attitude parameter control unit 28 and theflow proceeds to step S94.

In step S94, the attitude parameter control unit 28 causes the searchunit 29 to search the vogue design information database 95, to determinewhether or not there is registered in the vogue design informationdatabase 95 any vogue design information which would restrict designaccording to attitude parameters received from the master manipulator 42_(k) in step S93.

In step S94, in the event that determination is made that there is novogue design information which would restrict design according toattitude parameters received from the master manipulator 42 _(k)registered in the vogue design information database 95, i.e., in theevent that the design according to the attitude parameters do notmarkedly deviate (unique design) from that which is en vogue,represented by the vogue design information, the flow proceeds to stepS99, where the attitude parameter control unit 28 supplies the attitudeparameters from the master manipulator 42 _(k) to the input/outputcontrol device 21, and the input/output control device 21 transmits theattitude parameters from the attitude parameter control unit 28 to thepseudo slave manipulator 41. Subsequently, the flow returns to step S92,and the same processing is repeated thereafter.

Accordingly, in this case, the movement of the virtual tip of the pseudoslave manipulator 41 follows the operations of the user U_(k) of themaster manipulator 42 _(k).

On the other hand, in step S94, in the event that determination is madethat there is vogue design information which would restrict designaccording to attitude parameters received from the master manipulator 42_(k) registered in the vogue design information database 95, i.e., inthe event that the design according to the attitude parameters deviate(unique design) from that which is en vogue, represented by the voguedesign information (in the event that the design is far-fetched in lightof that which is currently en vogue), the flow proceeds to step S95,where the attitude parameter control unit 28 controls the attitudeparameter registration unit 22 such that the design information from themaster manipulator 42 _(k) representing unique design is registered inthe unique design information database 94, and the flow proceeds to stepS96.

In step S96, the updating unit 26 determines whether the frequency ofthe unique design information registered in the unique designinformation database 94 in the immediately-prior step S95 (hereafterreferred to as immediately-prior unique design information, asappropriate), is greater than a predetermined threshold value.

In step S96, in the event that the frequency of the immediately-priorunique design information is determined not to be greater than thepredetermined threshold value, i.e., in the event that operations forcarrying out unique design which the immediately-prior unique designinformation indicates, is not being performed very much by one or moreusers, the flow proceeds to step S97, and the attitude parameter controlunit 28 corrects the attitude parameters from the master manipulator 42_(k) based on the vogue design information for restricting the designaccording to those attitude parameters, so as to come within the rangeof restriction under the vogue design information. The flow thenproceeds to step S99, where the attitude parameter control unit 28transmits the corrected attitude parameters to the pseudo slavemanipulator 41 via the input/output control device 21, the flow returnsto step S92, and subsequently, the same processing is repeated.

Accordingly, in this case, the movement of the virtual tip of the pseudoslave manipulator 41 follows a movement wherein the operations of theuser U_(k) of the master manipulator 42 _(k) have been corrected basedon the vogue design information, i.e., wherein the outlandish design(unique design) which the user was attempting is corrected so as to benot too far removed from the design en vogue.

On the other hand, in step S96, in the event that the frequency of theimmediately-prior unique design information is determined to be great,i.e., in the event that operations for carrying out unique design whichthe immediately-prior unique design information indicates, is beingperformed frequently by one or more users, and as a result, the designcurrently en vogue is changing into a design which the immediately-priorunique design information indicates, the flow proceeds to step S98,where the updating unit 26 reads out the immediately-prior unique designinformation from the unique design information database 94, andtransfers this to the vogue design information database 95 to be storedas vogue design information.

Subsequently, the flow proceeds to step S99, the attitude parametercontrol unit 28 transmits the attitude parameters from the mastermanipulator 42 _(k) to the input/output control device 21, and theinput/output control device 21 transmits the attitude parameters fromthe attitude parameter control unit 28 to the pseudo slave manipulator41. Subsequently the flow returns to step S92, and the same processingis repeated thereafter.

Accordingly, in this case, the movement of the virtual tip of the pseudoslave manipulator 41 follows the operations of the user U_(k) of themaster manipulator 42 _(k).

That is to say, in the event that the design according to the attitudeparameters from the master manipulator 42 _(k) is a unique design whichdeviates from the design that is en vogue as indicated by the voguedesign information (in the event that the design is far-fetched in lightof that which is currently en vogue), operations to perform such aunique design are corrected based on the vogue design information atfirst. However, even if the deign is a unique design at first, in theevent that many operations are made to carry out that unique design, theunique design is assumed to have been accepted by many, such that thedesign currently en vogue has shifted to that unique design.Accordingly, in this case, the unique design is additionally stored inthe vogue design information database 95 as vogue design information,and further, operations of users for carrying out this unique design areno loner restricted (corrected) by vogue design information.

Consequently, while the virtual tip of the pseudo slave manipulator 41does not follow operations for carrying out the unique design, thevirtual tip of the pseudo slave manipulator 41 comes follow operationsfor carrying out the unique design as the unique design comes to beaccepted by many. That is, in the event that a PDA design such as shownin FIG. 18A is en vogue, attempts to operate the master manipulator 42so as to carry out design of a PDA such as shown in FIG. 18B arerestricted at first, but as many such operations are carried out, therestrictions not made any more, so that a PDA design such as shown inFIG. 18B becomes possible.

As described above, with the SHARN design system shown in FIG. 14, thevogue design information database 95 is updated based on designsperformed by each of the users U₁ through U_(K) operating the mastermanipulators 42 ₁ through 42 _(K) (additional storage of unique designinformation in the case described above), but this SHARN design systemis further arranged such that the vogue design information database 95is updated based on evaluations of other general users.

Accordingly, the processing of updating the vogue design informationdatabase 95 based on evaluations of other general users will bedescried, with reference to the flowchart in FIG. 22.

First, in step S111, the design evaluation unit 93 collects evaluationinformation evaluating designs according to the design informationregistered in the design information database 92 by the designinformation registration unit 91.

That is to say, the design evaluation unit 93 collects as evaluationinformation, via the network 43, the results of popularity votes cast bygeneral users regarding design information, by means of a web pagedisclosing the design information stored in the design informationdatabase 92, for example. Also, the design evaluation unit 93 collectsthe number of PDA units determined according to design informationstored in the design information database 92, as evaluation information,via the network 43.

The flow then proceeds to step S112, where the design evaluation unit 93calculates the evaluation values for each design information stored inthe design information database 92, based on the evaluation informationcollected in step S111, and updates the design information database 92based on the evaluation values.

That is to say, the design information database 92 has designinformation stored in a manner correlated with the evaluation valuesobtained in the previous updating processing (hereafter referred to asprevious evaluation value, as appropriate), and in step S112, theevaluation values (previous evaluation values) correlated to each designinformation, are updated according to evaluation values newly calculatedwith regard to the design information (hereafter referred to as currentevaluation value, as appropriate).

Now, the evaluation value is a value of 0 or greater, and the greaterthe value is, the higher the evaluation level is.

Subsequently, the flow proceeds to step S113, where the updating unit 26determines whether or not there is design information with an evaluationvalue of a predetermined threshold value ε₁ or greater with a rate ofchange in the evaluation value equal to or greater than the thresholdvalue+ε₂ in the design information database 92.

In step S113, in the event that determination is made that there isdesign information with an evaluation value of a predetermined thresholdvalue ε₁ or greater with a rate of change in the evaluation value equalto or greater than the threshold value+ε₂ in the design informationdatabase 92, that is to say, in the event that there is designinformation wherein the evaluation value itself is great, and further,the change in evaluation value (the increase) is also great (hereafterreferred to as popular design information, as appropriate), the flowproceeds to step S114, and the updating unit 26 determines whether thepopular design information is already registered as vogue designinformation in the vogue design information database 95.

In the event that determination is made in step S114 that the populardesign information is already registered as vogue design information inthe vogue design information database 95, the flow returns to step S111,and the same processing is repeated.

In the event that determination is made in step S114 that the populardesign information has not yet been registered as vogue designinformation in the vogue design information database 95, the flowproceeds to step S115, where the updating unit 26 reads out the populardesign information from the design information database 92, and suppliesand additionally stores this to the vogue design information database 95as vogue design information, thereby updating the vogue designinformation database 95, and the flow returns to step S111.

In step S113, in the event that determination is made that there is nodesign information with an evaluation value of a predetermined thresholdvalue ε₁ or greater with a rate of change in the evaluation value equalto or greater than the threshold value+ε₂ in the design informationdatabase 92, that is to say, in the event that there is no populardesign information, the flow proceeds to step S116, where the updatingunit 26 determines whether there is design information registered in thedesign information database 92 of which the rate of change of theevaluation value has been 0 or less of a predetermined threshold valueε₃ number of times or more.

In the event that determination is made in step S116 that there isdesign information of which the rate of change of the evaluation valuehas been 0 or less of a predetermined threshold value ε₃ number of timesor more, i.e., wherein the evaluation value does not change or onlydecreases (hereafter referred to as unpopular design information, asappropriate), the flow proceeds to step S117, where the updating unit 26determines whether there is any unpopular design information registeredin the vogue design information database 95 as vogue design information.

In the event that determination is made in step S117 that there isunpopular design information registered in the vogue design informationdatabase 95 as vogue design information, the flow proceeds to step S118,where the updating unit 26 deletes the unpopular design information asvogue design information from the vogue design information database 95,thereby updating the vogue design information database 95, and the flowreturns to step S111.

Also, in the event that determination is made in step S117 that there isno unpopular design information registered in the vogue designinformation database 95 as vogue design information, step S118 isskipped, and the flow returns to step S111.

As described above, with the SHARN design system shown in FIG. 14, thevogue design information in the vogue design information database 95 inthe super designer database unit 44 is updated into a better one (designmatching what is currently en vogue) by attitude parameters based onoperations of each of the master manipulators 42 ₁ through 42 _(K).Further, vogue design information of the vogue design informationdatabase 95 in the super designer database unit 44 is updated in to abetter one by evaluations made by general users, as well. Further, withthe super designer database unit 44, attitude parameters to be providedto the pseudo slave manipulator 41 (reference attitude parameters) aregenerated from the attitude parameters based on the operations of themaster manipulator 42 _(k) and the vogue design information, and at thepseudo slave manipulator 41, the attitude of the virtual tip iscontrolled according to the reference attitude parameters, wherebydesign is made, while on the other hand, a load is placed on the userU_(k) operating the operating unit 9 so as to correct, as if it were,the offset between the operations of the user U_(k) and the referenceattitude parameters.

As a result, at the pseudo slave manipulator 41, design is executed inaccordance with what is en vogue, in a manner of speaking, based on thedesign tastes of the user U_(k), and vogue design information which ismade up and is updated according to the design tastes of many otherusers.

Also, with the master manipulator 42 _(k), the operations of the userU_(k) is corrected based on the vogue design information, so improvementof design skills of the user U_(k) can be realized in a short time.

Now, with the SHARN design system shown in FIG. 14, the vogue designinformation of the super designer database unit 44 is updated into abetter one based on the operations of each of the master manipulators 42₁ through 42 _(K) as shown in FIG. 23, and further, at the mastermanipulators 42 ₁ through 42 _(K), feedback is received as if it wereregarding vogue design information, so that the user operations arecorrected so as to match vogue design information.

Accordingly, at each of the master manipulators 42 ₁ through 42 _(K),high functionality of improving the design skills of the users U₁through U_(K) can be realized, which could not be realized individually,due to the master manipulators 42 ₁ through 42 _(K), and the superdesigner database unit 44, surrounded by the dotted line in FIG. 23,acting as if they were a single apparatus, with the vogue designinformation of the super designer database unit 44 being updated into abetter one based on the operations of each of the master manipulators 42₁ through 42 _(K), whereby each of the master manipulators 42 ₁ through42 _(K) share the processing of updating the vogue design informationinto a better one through collaboration.

Conversely, in the event that the super designer database unit 44 onlystores the operation contents of each of the master manipulators 42 ₁through 42 _(K), and gives feedback to the master manipulators 42 ₁through 42 _(K), this is only simple sharing of the processing of themaster manipulators 42 ₁ through 42 _(K) supplying the operations madeby users to the super designer database unit 44, and accordingly, thiscannot realize the high functionality at each of the master manipulators42 ₁ through 42 _(K) of the design skills of the users U₁ through _(K)being improved at each of the master manipulators 42 ₁ through 42 _(K).

Next, FIG. 24 illustrates a configuration example of a SHARN movingentity system serving as a moving entity system to which the SHARNsystem has been applied.

This SHARN moving entity system is configured of a great number ofmoving entities (e.g., automobiles or the like) 101 ₁, 101 ₂, . . . ,all-directional cameras 102 _(k) mounted on each moving entity 101 _(k),and a network 104.

An automobile service as a moving body 101 _(k) moves over roads or thelike by acting in response to operations made by a user U_(k) driving.An all-directional camera 102 _(k) provided to the moving entity 101_(k) takes (detects) all-directional images centered on itself.Accordingly, an all-directional camera 102 _(k) on a moving entity 101_(k) capable of direct-line sight of a subject 103 can take imagesincluding the subject 103 even without giving consideration to thedirection in which the subject 103 exists. The network 104 is a networkenabling wireless communication between arbitrary moving entities 101_(k) and 101 _(k′).

Now, in the following, the all-directional image obtained by theall-directional camera 102 _(k) centered on itself, will be referred toas all direction image, as appropriate.

The moving entity 101 _(k) communicates with other moving entities 101_(k′) via the network 104. The moving entity 101 _(k) receives imagedata taken with other moving entities 101 _(k′) via the network 104, andprocesses that image data, thereby generating added-value added imagedata.

Now, with the embodiment shown in FIG. 24, the moving entity 101 _(k)corresponds to the operating unit 1003, actuator 1005, synthesizingintegrating unit 1002, driving control unit 1004, and signal processingunit 1006 in FIG. 1 (and FIG. 2), the all-directional cameral 102 _(k)corresponds to the sensor 1007 in FIG. 1, and the network 104corresponds to the network 1001 in FIG. 1.

Note that communication with a moving entity 101 _(k′) is not restrictedto only the moving entity 101 _(k), and an information processing device105 serving as a fixed station can also perform this. The informationprocessing device 105 which is the fixed station can receive image datataken with other moving entities 101 _(k′) via the network 104, andprocesses that image data, thereby generating added-value added imagedata, as with the moving entity 101 _(k).

Next, the functions of the SHARN moving entity system shown in FIG. 24will be briefly described with reference to FIG. 25.

With the SHARN moving entity system, as a first function, a movingentity 101 ₈ for example, which is a desired moving entity, is specifiedby an unshown moving entity 101 _(k), so as to obtain theall-directional image data taken by the all-directional camera 102 ₈ atthe location of the moving entity 101 ₈, via the network 104.

Also, as a second function, the moving entity 101 _(k) can specify amoving entity situated in a desired line-of view direction v with regardto a subject 103 (the moving entity 101 ₇ in FIG. 25), and obtain theall-directional image data taken by the all-directional camera 102 ₇ atthe location of the moving entity 101 ₇, via the network 104. In thiscase, the all-directional image data includes an image viewing thesubject 103 from the predetermined line-of view direction v.

Further, as a third function, the moving entity 101 _(k) can obtainall-directional image data including images viewing the subject 103 froma great number of viewpoint directions, taken by each of multiple movingentities (in FIG. 25, the moving entities 101 ₁, 101 ₂, 101 ₃, 101 ₄,101 ₅, and 101 ₆, positioned on the dotted line circle surrounding thesubject 103), via the network 104, and process the multiple sets ofall-directional image data, thereby instantaneously obtain an imageviewing the subject 103 while changing viewpoints (an image as if thesubject 103 had been circled while being taken) (hereafter referred toas viewpoint-variable image as appropriate).

Now, the third function is realized by sharing processing for obtainingthe viewpoint-variable image with the moving entities 101 ₁, 101 ₂, 101₃, 101 ₄, 101 ₅, and 101 ₆, through collaboration, a point which will bedescribed in detail later.

Next, FIG. 26 illustrates a configuration example of a moving entity 101_(k).

The all-directional camera 102 _(k) is configured of an image-takingdevice 111 and an A/D (Analog/Digital) converting unit 112.

The image-taking device 111 takes (photographs) all-directional imagesaccording to control signals supplied from the control unit 118 via abus Bus1, and supplies the analog image signals obtained as a resultthereof to the A/D converting unit 112. Also, the image-taking device111 supplies zoom information and the like relating to zooming at thetime of taking the all-directional image, to a scaling unit 113.

The A/D converting unit 112 performs A/D conversion of the analogsignals from the image-taking device 111, and accordingly obtainsdigital all-directional image data, which is supplied to the scalingunit 113.

The scaling unit 113 converts the scale of the all-directional imagedata supplied from the all-directional camera 112 _(k) (the A/Dconverting unit 112 thereof) based on zoom information supplied from thesame all-directional camera 112 _(k) (the image-taking device 111thereof).

That is, the scaling unit 113 converts the all-directional image datataken at an arbitrary zoom magnification at the all-directional camera102 _(k) into that taken with a predetermined zoom magnification.

The all-directional image data processed at the scaling unit 113 issupplied to an image database 114, and the image database 114 stores theall-directional image data from the scaling unit 113 so as to becorrelated with necessary information. Also, the image database 114 alsostores (registers) image data and the like supplied from the controlunit 118 via the bus Bus1, as described later.

An operating unit 115 is operated for inputting predeterminedinformation (e.g., the zoom magnification of the image-taking device 111or the like) or commands, and the input is processed by being suppliedto the control unit 118 via the bus Bus1. Also, the operating unit 115also includes the steering wheel, accelerator, brake, etc., forcontrolling the moving body 101 _(k) as well.

A transmission unit 116 transmits the information supplied via the busBus1 via the network 104 (FIG. 24). A receiving unit 117 receives theinformation transmitted via the network 104, and outputs onto the busBus1.

The control unit 118 is connected to busses Bus1 and Bus2, and controlsthe blocks connected to the busses Bus1 and Bus2.

A display image processing unit 119 performs processing of the imagedata supplied via the bus Bus2 so as to generate an image to bedisplayed on a display unit 112, and outputs to the bus bus2.

A comparing unit 120 makes reference to a storage unit 121 and positiondetecting unit 123 via the bus Bus2, compares the information stored inthe storage unit 121 and information output from the position detectingunit 123, and notifies the control unit 118 of the comparison resultsvia the bus Bus2.

The storage unit 121 temporarily stores the information and the likesupplied from the control unit 118 via the bus Bus2.

A display unit 122 is configured of an LCD (Liquid crystal Display) orCRT (Cathode Ray Tube) or the like for example, and displays imagesaccording to image data supplied via the bus Bus2.

The position detecting unit 123 is configured of a GPS (GlobalPositioning System) or the like for example, and detects the currentlocation of the moving entity 101 ₁, and outputs to the bus Bus2.

A map database 124 stores map data which is electronic map data, and isreferred to by other blocks via the bus Bus2.

A coordinates computing unit 125 computes predetermined coordinates andso forth according to control signals supplied from the control unit 118via the bus Bus2.

A cropping unit 126 crops out a predetermined region from image datasupplied via the bus Bus2 under the control of the control unit 118, andoutputs onto the bus Bus2.

An image conversion unit 127 converts the image data supplied via thebus Bus2 under control of the control unit 118 into image data fromwhich later-described distortion has been removed, and outputs onto thebus bus2.

Next, FIG. 27 illustrates a configuration example of the image-takingdevice 111 shown in FIG. 26.

The image-taking device 111 has a configuration wherein a condensingunit 202 is supported by a supporting unit 201 disposed above animage-taking unit 203, so as to be able to take images in 360°directions around the condensing unit 202.

The supporting member 201 is formed of a transparent material with lowreflectivity such as glass or the like, and is a spherical face with thelater-described focal point of the condensing unit 202 as the centerthereof such that the incident light is orthogonal to the glass face, inorder to minimize refraction of light at the glass face.

The condensing unit 202 has a hyperboloidal form, which reflects thesurrounding light cast in through the supporting member 201 in thedirection of the image-taking unit 203.

The image-taking unit 203 is configured of an electro-optical convertingdevice such as a CCD or the like, for example, and converts light casttherein into image signals as electric signals. The image signalsobtained at the image-taking unit 203 are supplied to the A/D conversionunit 112 shown in FIG. 26 as the output of the image-taking device 111.

Now, the configuration of condensation principle of the image-takingdevice 111 making up the all-directional camera 102 _(k) is disclosedin, for example, “Development of All-Directional Visual Sensor forMoving Robot”, Automation Technology Vol. 29, No. 6 (1997) (hereafterreferred to as Document 1 as appropriate), so the description made herewill be brief.

FIG. 28 illustrates an example of all-directional image data taken bythe image-taking device 111 shown in FIG. 27.

The all-directional image data is sectioned by the perimeters F1 and F2of two circles centered on a center point C1. Note that in FIG. 28, theperimeter F1 has a greater radius than the perimeter F2.

That is to say, the all-directional image data is configured of a region(inner region) R3 between a region R1 which is within the largerperimeter F1 and outside the smaller perimeter F2, and a region R2outside the larger perimeter F1.

Of the three regions R1 through R3, the image data corresponding to thesurrounding light reflected at the condensing unit 202 is taken in theregion R1, and the portion outside of the condensing unit 202 is takenin the region R2. The image-taking unit 203 itself is taken in theregion R3.

Further, with the embodiment shown in FIG. 28, multiple buildings havebeen taken in the region R1, with the tops of the buildings toward thedirection closer to the perimeter F1 and the bottoms of the buildingstoward the direction closer to the perimeter F2.

Next, the hyperboloid making up the condensing unit 202 shown in FIG. 27will be described with reference to FIG. 29.

The condensing unit 202 is configured of part of a hyperboloid, and isformed by forming a mirror face on a convex face of the convex tip of ahyperboloid obtained by cutting a hyperboloid along a planeperpendicular to the axis of the hyperboloid.

A hyperboloid of two sheets, which is a type of hyperboloid, can be usedfor the condensing unit 202, and as shown in FIG. 29, a hyperboloid oftwo sheets is a curved surface obtained by rotating a hyperbolic curveon the axis (Z axis), and is configured of a downward convex hyperboloidH1 existing in a Z>0 region and an upward convex hyperboloid H2 existingin a 0>Z region.

Of the hyperboloid of two sheets, H1 and H2, the hyperboloid H1 in theZ>0 region is used for the condensing unit 202. Note that hereafter, theZ axis will be referred to as the center axis of the hyperbolic curve,or simply as axis, whenever appropriate.

In the three-dimensional orthogonal coordinates system of X, Y, and Z,shown in FIG. 29, the hyperboloid of two sheets, H1 and H2, arerepresented as shown in Expression (2).

$\begin{matrix}{{\frac{X^{2} + Y^{2}}{a^{2}} - \frac{Z^{2}}{b^{2}}} = {- 1}} & (2)\end{matrix}$

Here, the constants a and b in Expression (2) are constants for definingthe shapes of the hyperboloids H1 and H2. That is to say, the constant brepresents the distance from the point of origin O to the intersectionbetween the hyperboloid H1 (H2) and Z axis. Also, the constant arepresents the radius inscribed by the intersecting line of a planewhich passes through that intersection and is parallel to the XY faceand asymptotic face A1 of the hyperboloid H1 (H2).

Further, in Expression (2), a constant c is a constant for defining theposition of the focal point of the hyperboloids H1 and H2. That is tosay, the position +c on the Z axis (0, 0, +c) is the focal point f1 ofthe hyperboloid H2, and the position −c on the Z axis (0, 0, −c) is thefocal point f2 of the hyperboloid H1.

The constants a, b, and c have a relation represented by the followingexpression.C=√{square root over (a ² +b ²)}  (3)

In the event that the condensing unit 202 is configured of a hyperboloidH1 such as described in FIG. 29, the image-taking unit 203 is positionedsuch that the center axis of the lens matches the Z axis, and such thatthe center of the lens matches the focal point f2 of the hyperboloid H1,as shown in FIG. 30.

Next, the all-directional image data output by the image-taking unit 203is such as shown with FIG. 28 described above; now, with regard to thisall-directional image data, a two-dimensional orthogonal coordinatessystem is defined wherein the point of origin O is at the upper leftportion thereof, and with the x axis from the left to right directionand the y axis from the top to bottom direction, as shown in FIG. 31.

In this case, with the coordinates of the center point C1 of the circlefor the perimeter F1 and the circle for the perimeter F2 as (X₀, Y₀),and the radius of the circle of the perimeter F1 as r_(F1) and theradius of the circle of the perimeter F2 as r_(F2), the point (x, y)making up the region R2 outside of the perimeter F1 is expressed byExpression (4), and the point (x, y) making up the region R3 inside ofthe perimeter F2 is expressed by Expression (5).(x−X ₀)²+(y−Y ₀)² >r _(F1) ²  (4)(x−X ₀)²+(y−Y ₀)² <r _(F2) ²  (5)

Before converting the scale of the all-directional data, the scalingunit 113 shown in FIG. 26 first converts the pixel values making up theregion R2 expressed by Expression (4) and the pixel values making up theregion R3 expressed by Expression (5) into, for example, 0.

Now, at the time of conversion of the pixel values, the coordinates (X₀,Y₀) of the center point C1, and the radii r_(F1) and r_(F2) arenecessary, and this information is arranged to be supplied to thescaling unit 113 from the image-taking device 111 along with the zoominformation. Note that the image-taking device 111 (FIG. 26) is arrangedso as to output to the scaling unit 113 information related to settingsof the image-taking device 111, such as the exposure time forimage-taking, the aperture, and so forth, besides the zoom information,the coordinates (X₀, Y₀) of the center point C1, the radii r_(F1) andr_(F2), as image taking information.

Here, let us assume that the exposure time and aperture are fixed, inorder to simplify description.

Now, with regard to the all-directional image data which is the imagedata of region R1 shown in FIG. 28 and FIG. 31, there are cases whereinonly part of the all-directional data is necessary, such as a portionwhere a desired subject is displayed, and so forth.

The cropping unit 126 shown in FIG. 26 is arranged to crop out a part ofthe all-directional image data, with the cropping being performed asshown in FIG. 32.

That is to say, the cropping unit 126 crops out the image data in afan-shaped region R11 which is surrounded by the two lines L_(c1) andL_(c2) in the radial direction which pass through the center point C1,and surrounded by the two perimeters F_(c1) and F_(c2) centered on thecenter point C1.

Next, the all-directional image data has distortion which becomesnarrower the closer to the center point C1, and conversely wider thefarther from the center point C1, as shown in FIG. 28 and FIG. 31. Thisis the same for the fan-shaped image data to be cropped out from theall-directional image data by the cropping unit 126, so such image datawith distortion needs to be converted into with no distortion, to bedisplayed on the display unit 122 shown in FIG. 26 or the like.

The distortion of the all-directional image data such as describedabove, output from the all-directional camera 102, came be consideredseparately regarding the circumferential direction of the circle passingthrough the center point C1, and distortion in the radial direction.

Now, first considering the distortion in the circumferential direction,the scale of the all-directional image data in FIG. 28 and FIG. 31 isconstant on the perimeter of a certain circle centered on the centerpoint C1, and accordingly, there is no distortion.

That is to say, FIG. 33 illustrates the hyperboloid H1 which is thecondensing unit 202 viewed from the Z-axial direction.

In FIG. 33, a ray r is cast in from a point P (X, Y, Z) on a subject,toward the Z axis, such that there is an angle θ as to the X axis (XZplane). The ray r is reflected at the hyperboloid H1, and reaches apoint p (x, y) on a photo-reception face (image face) A3 of theimage-taking unit 203 shown in FIG. 34 described later. In this case,the point p (x, y) assumes the same angle θ as to the X axis (XZ plane)as the ray r as to the X axis.

Also, considering a ray r′ toward the Z axis assuming an angle θ′ as tothe X axis, this ray r′ is also reflected at the hyperboloid H1, andreaches a point p′ (x′, y′) on the photo-reception face (image face) A3.In this case as well, the point p′ (x′, y′) assumes the same angle θ′ asto the X axis (XZ plane) as the ray r′ as to the X axis.

Also, in the event that the elevation angle (described later) of therays r and r′ are the same as to the Z axis, the rays r and r′ arereceived at different points on the photo-reception face A3, at the samedistance r_(p) from the Z axis.

Thus, each of the multiple rays with the same elevation angle as to theZ axis are points at the same distance from the Z axis on thephoto-reception face A3, and are received at points where the rays and Xaxis form the same angle with the X axis.

Accordingly, multiple rays with the same elevation angle as to the Zaxis and the same angle as to the X axis are cast onto the perimeter ofthe photo-reception face A3 centered on the Z axis at constantintervals, so, with the all-directional image data, there is nodistortion in the circumferential direction of a circle centered on thecenter point C1 which is the intersection of the photo-reception face A3and the Z axis.

On the other hand, on a straight line in the radial direction of theall-directional image data shown in FIG. 28 and FIG. 31 passing throughthe center point C1, the closer to the center point C1 the smaller thescale is, and the farther from the center point C1 the greater the scaleis, so the all-directional image data has distortion in the radialdirection.

That is to say, FIG. 34 shows the hyperboloid H1 as the condensing unit202 as viewed from the X-axial direction.

In FIG. 34, a ray r1 cast in toward a focal point f1 at an elevationangle of 0° as to the Z axis is reflected at the hyperboloid H1, travelstoward a focal point f2, and is received at the photo-reception face(image face) A3 of the image-taking unit 203.

Now, the elevation angle as to the Z axis means an angle created betweena plane which passes through the focal point f1 on the Z axis and isparallel with the XY plane, and a ray cast in toward the focal point f1.Also, with the focal distance of the image-taking device 111 as f, thephoto-reception face A3 of the image-taking unit 203 is positioned adistance f away from the focal point f2 along the Z axis in thedirection of the point of origin O.

A ray r2 cast in toward the focal point f1 at an elevation angle of Δd°as to the Z axis is reflected at the hyperboloid H1, travels toward thefocal point f2, and is received at the photo-reception face (image face)A3 of the image-taking unit 203. A ray r3 cast in toward the focal pointf1 at an elevation angle of 2Δd° as to the Z axis is reflected at thehyperboloid H1, travels toward the focal point f2, and is received atthe photo-reception face (image face) A3 of the image-taking unit 203.Rays r4, r5, and r6, with elevation angles increasing by Δd° each arealso received at the photo-reception face A3 in the same way.

Thus, the intervals between points received at the photo-reception faceA3 for each of the rays r1 through r6 having elevation angles differingby Δd° are not at equal intervals as shown in the enlarged drawing inFIG. 34, but rather the closer the points are to the Z axis the narrowthe intervals are (the farther the points are from the Z axis the widerthe intervals are). That is to say, the greater the elevation angle ofrays are, the narrower the intervals therebetween are (the smaller theelevation angle of rays are, the wider the intervals therebetween are).

Since the intervals between points received on the photo-reception faceA3 are not equal with regard to rays with differing elevation angles, asdescribed above, there is distortion in the all-directional image datain the radial direction.

Accordingly, to remove the distortion of the all-directional image datain the radial direction, the intervals between the photo-receptionpoints on the photo-reception face A3 should be made equal for rays withequal angles for the elevation angle.

Accordingly, let us now consider a ray r which is cast in toward thefocal point f1 from a point P (X, Y, Z) on a subject at an elevationangle of α° as shown in FIG. 35.

This ray r is reflected at the hyperboloid H1 serving as the condensingunit 202, travels toward the focal point f2, and is received at thephoto-reception face A3, in the same way as described with FIG. 34.

In this case, the angle which the ray r heading toward the focal pointf2 forms with a plane which passes through the focal point f2 and isparallel to the XY plane is γ°. Further, in the two-dimensionalorthogonal coordinates system defined by the X axis and Y axis, thepoint p where the ray r is received by the photo-reception face A3(photo-reception point) is represented with the coordinates (x, y), andthe distance between the point p (x, y) and the Z axis is represented byR.

The angle α in FIG. 35 can be expressed with the following Expression,by obtaining the coordinates of the point where the ray r is reflectedat the hyperboloid H1 expressed by Expression (2).

$\begin{matrix}{\alpha = {\tan^{- 1}\frac{{\left( {b^{2} + c^{2}} \right)\;\sin\;\gamma} - {2{bc}}}{\left( {b^{2} - c^{2}} \right)\;\cos\;\gamma}}} & (6)\end{matrix}$

From Expression (6), tan α can be obtained by the following Expression.

$\begin{matrix}{{\tan\;\alpha} = \frac{{\left( {b^{2} + c^{2}} \right)\;\sin\;\gamma} - {2{bc}}}{\left( {b^{2} - c^{2}} \right)\;\cos\;\gamma}} & (7)\end{matrix}$

Here, cos γ is defined as in the next Expression.cos γ=X  (8)

Substituting Expression (8) into Expression (7) yields the followingExpression.{(b ² −C ²)tan α}·X=(b ² +c ²)√{square root over (1−X ²)}−2bc  (9)

Expression (9) is modified to yield Expression (10).

$\begin{matrix}{{{\frac{\left( {b^{2} - c^{2}} \right)\;\tan\;\alpha}{b^{2} + c^{2}} \cdot X} + \frac{2{bc}}{b^{2} + c^{2}}} = \sqrt{1 - X^{2}}} & (10)\end{matrix}$

On the other hand, A and B are each defined as in Expressions (11) and(12).

$\begin{matrix}{A = \frac{\left( {b^{2} - c^{2}} \right)\;\tan\;\alpha}{b^{2} + c^{2}}} & (11) \\{B = \frac{2{bc}}{b^{2} + c^{2}}} & (12)\end{matrix}$

Substituting Expressions (11) and (12) into Expression (10) yieldsExpression (13).AX+B=√{square root over (1−X ²)}  (13)

Expression (13) can be formed as the following quadratic equationregarding X.(A ²+1)X ²+2ABX+B ²−1=0  (14)

Solving the quadratic equation of Expression (14) yields the followingExpression.

$\begin{matrix}{X = \frac{{- {AB}} + \sqrt{A^{2} - B^{2} + 1}}{A^{2} + 1}} & (15)\end{matrix}$

Accordingly, the angle γ can be obtained from Expression (8) from thefollowing Expression.γ=cos⁻¹(X)  (16)

Note however that in Expression (16), X is defined in Expression (15),and also, A and B in Expression (15) are defined in Expressions (11) and(12).

Also, the angle γ can be obtained from the coordinates of the point pshown in FIG. 35, from the following Expression.

$\begin{matrix}{\gamma = {\tan^{- 1}\left( \frac{f}{R} \right)}} & (17)\end{matrix}$

On the other hand, the distance R between the Z axis and point p (x, y)in FIG. 35 is expressed as √{square root over ((x²+y²))}, andaccordingly Expression (17) can be modified as in the followingExpression.

$\begin{matrix}{R = {\sqrt{x^{2} + y^{2}} = \left( \frac{f}{\tan\;\gamma} \right)}} & (18)\end{matrix}$

From the above, obtaining the angle γ with Expression (16) and using theangle γ to calculate Expression (18) allows calculation of the distanceR from the Z axis of the point p (x, y) where the ray r, cast in towardthe focal point f1 at an elevation angle of α° as to the Z axis isreceived at the photo-reception face A3.

Accordingly, the distortion of the all-directional image data in theradial direction can be removed by dividing a predetermined range ofelevation angle as to the Z axis (e.g., 0° through 80° or the like) intoequal angles Δd, calculating distances R_(p1), R_(p2), R_(p3), . . .from the Z axis for each of the points p1, p2, p3 . . . where theincident rays at the elevation angles Δd, 2Δd, 3Δd, . . . are receivedon the photo-reception face A3, and the all-directional image dataoutput by the image-taking device 111 is converted so that thedifference in distance between the adjacent points, |R_(p1)−R_(p2)|,|R_(p2)−R_(p3)|, . . . is constant.

Note that the conversion of the all-directional image data is performedby the image conversion unit 127 shown in FIG. 26, to remove thisdistortion.

That is to say, the image conversion unit 127 receives theall-directional image data with the distortion in the radial directiondescribed above (hereafter referred to as distorted all-directionalimage data, as appropriate) from the control unit 118 as shown in FIG.36, and also receives the center point C1, and the radius of the circleof the perimeter F1 as r_(F1) and the radius of the circle of theperimeter F2 as r_(F2).

The image conversion unit 127 then draws multiple concentric circleswithin the range R1 between the perimeter F1 of the circle with theradius r_(F1) centered on the center point C1 and the perimeter F2 ofthe circle with the radius r_(F2) centered on the center point C1,having the above described radii R_(p1), R_(p2), R_(p3), . . . .Further, the image conversion unit 127 draws multiple straight linespassing through the center point C1, equally dividing the multipleconcentric circles. Accordingly, the image conversion unit 127 yields agreat number of intersections formed between the multiple concentriccircles and the multiple lines in the region R1.

Now, the great number of intersections formed between the multipleconcentric circles and the multiple lines will hereafter be referred toas conversion object points, as appropriate.

From the above, the conversion object points are placed so that theadjacent ones are each at equal intervals, whereby the all-directionalimage data with distortion can be converted into image data from whichthe distortion has been removed.

Accordingly, let us now consider converting the image data of the regionR11 in FIG. 36 having the same fan-shaped form as shown in FIG. 32, tothat with no distortion.

Within the region R11 there are a great number of conversion objectpoints, which are the intersections between each of the multipleconcentric arcs F11, F12, F13, F14, F15, and F16, centered on the centerpoint C1, and each of the multiple straight lines L1, L2, L3, L4, L5,and L6, as shown in FIG. 37A. The image conversion unit 127 arrays thegreat number of conversion object points existing within the region R11such that the adjacent ones are at equal intervals, as shown in FIG.37B.

That is to say, the image conversion unit 127 assumes equidistantlyplaced horizontal straight lines L21, L22, L23, L24, L25, and L26,corresponding to each of the arcs F11, F12, F13, F14, F15, and F16,shown in FIG. 37A, and equidistantly placed vertical straight lines L11,L12, L13, L14, L15, and L16, corresponding to each of the multiplestraight lines L1, L2, L3, L4, L5, and L6, as shown in FIG. 37A, andconverts (assigns) corresponding object points on the intersectionpoints between each of the horizontal straight lines L21 through L26 andeach of the vertical straight lines L11 through L16.

Accordingly, in FIG. 37A for example, the conversion object point p11which is the intersection between the arc F14 and the straight line L5is converted into the intersection p12 between the straight line L24corresponding to the arc 14 and the straight line L15 corresponding tothe straight line L5.

Now, as described above, the image conversions unit 127 converts theimage data having the distortion in the radial direction into image datafrom which the distortion has been removed, by performing conversion(changing) of the position of the conversion object points, with theconversion object points being the intersections between circlescentered on the center point C1 and straight lines passing through thecenter C1, as described above. Accordingly, the conversion object pointsdo not necessarily match the pixel centers of the pixels making up thephoto-reception face A3 (FIG. 34, FIG. 35) of the image-taking unit 203(FIG. 27), and rather, generally, these often do not match.

In this way, the conversion object points are generally off from thepixel centers, so there is the need for the image conversion unit 127 toobtain the pixel values at the conversion object points beforeconverting the positions of the conversion object points.

Accordingly, the image conversion unit 127 is arranged so as to obtainthe pixel values at the conversion object points as follows, forexample.

That is to say, let us say that the conversion object point p11 shown inFIG. 37 is within the range of a rectangle (square) having vertexes ofC_(P1), C_(P2), C_(P3), and C_(P4), which are pixel centers of the fourpixels P1, P2, P3, and P4, which are adjacent on two-by-two fashion onthe photo-reception face A3 as shown in FIG. 38. In the embodiment shownin FIG. 38, the distance between vertically and horizontally adjacentpixel centers is 1.

In the event that the conversion object point p11 is a point which is tothe right of the center pixel C_(P1) by a distance α and down by adistance β, the image conversion unit 127 takes a pixel value S_(s)obtained with the following Expression, for example, as the pixel valueof the conversion object point p11.S _(s) ={S _(a)·(1−α)+α·S _(b)}(1−β)+β·{S _(c)·(1−α)+α·S _(d)}  (19)

In Expression (19), S_(a), S_(b), S_(c), and S_(d) each represent thepixel values of the pixels P1, P2, P3, and P4. Also, in expression (19),α and β are values within the range of 0 or greater and 1 or smaller,corresponding to the position of the conversion object point p11.

Next, the scaling unit 113 shown in FIG. 26, as described above,converts the all-directional image data taken with the all-directionalcamera 102 _(k) with an arbitrary zoom magnification, into that takenwith a predetermined zoom magnification. This zoom magnificationconversion may be realized by simple pixel interpolation and thinning,and necessary filtering such as low-pass filter and the like, but inorder to improve magnification, the class classification adaptationprocessing already proposed by the present applicant, for example, maybe used instead of simple pixel interpolation.

Class classification adaptation processing is made up of classclassification processing and adaptation processing, wherein the data isdivided into classes based on the nature thereof in the classclassification processing, and adaptation processing is implemented foreach class.

Now, the adaptation processing will be described, with regard to anexample of a case of converting an image of low magnification (hereafterreferred to as low-magnification image, as appropriate) into an image ofhigh magnification (hereafter referred to as high-magnification image,as appropriate).

In this case, with the adaptation processing, linear combination ofpixels making up a low-magnification image (hereafter referred to aslow-magnification pixels, as appropriate) and predetermined tapcoefficients yields prediction values of pixels of a high-magnificationimage wherein the spatial resolution of the low-magnification image hasbeen improved, thereby obtaining an image wherein the magnification ofthe low-magnification image has been raised.

Specifically, for example, let us consider taking a high-magnificationimage as tutor data and a low-magnification image wherein the resolutionof the high-magnification image has been deteriorated as student data,and obtaining a prediction value E[y] of a pixel value y of a pixelmaking up the high-magnification image (hereafter referred to ashigh-magnification pixel, as appropriate), by a linear combination modelstipulated by linear combination of the pixel values x₁, x₂, . . . ofseveral low-magnification pixels (pixels making up a low-resolutionimage) and predetermined tap coefficients w₁, w₂, . . . . In this case,the prediction value E[y] can be expressed with the followingExpression.E[y]=w ₁ x ₁ +w ₂ x ₂+  (20)

In order to generalize Expression (20), defining a matrix W of a groupof tap coefficients w_(j), a matrix X of a group of student data x_(ij),and a matrix Y′ of a group of prediction values E[y_(j)], as

$X = \begin{pmatrix}x_{11} & x_{12} & \ldots & x_{1J} \\x_{21} & x_{22} & \ldots & x_{2J} \\\ldots & \ldots & \ldots & \ldots \\x_{I\; 1} & x_{I\; 2} & \ldots & x_{IJ}\end{pmatrix}$ ${W = \begin{pmatrix}\begin{matrix}\begin{matrix}W_{1} \\W_{2}\end{matrix} \\\ldots\end{matrix} \\W_{J}\end{pmatrix}},{Y^{\prime} = \begin{pmatrix}\begin{matrix}\begin{matrix}{E\left\lbrack y_{1} \right\rbrack} \\{E\left\lbrack y_{2} \right\rbrack}\end{matrix} \\\ldots\end{matrix} \\{E\left\lbrack y_{I} \right\rbrack}\end{pmatrix}}$yields the following observational equationXW=Y′  (21)

Here, the component x_(ij) of the matrix X means the j'th student datain the i'th student data group (the group of student data used forprediction of the i'th tutor data y_(i)), and the component w_(j) of thematrix W means the tap coefficient of which the product with the j'thstudent data in the student data group is computed. Also, y_(i) meansthe i'th tutor data, and accordingly, E[y_(i)] means the predictionvalue of the i'th tutor data. Note that the y to the left side ofExpression (20) has the suffix i of the matrix Y component y_(i)omitted, and also, the x₁, x₂, . . . to the right side of Expression(20) have the suffix i of the matrix X component x_(ij) omitted.

Let us consider obtaining a prediction value E[y] close to the pixelvalue y of the high-magnification pixel by applying the leas-squaremethod to the observational equation of Expression (21). In this case,defining a matrix Y of a group of true pixel values y ofhigh-magnification pixels to serve as the tutor data, and a matrix E ofa group of the residual error e of the prediction value E[y] as to thepixel value y of the high-magnification pixel, as

${E = \begin{pmatrix}e_{1} \\e_{2} \\\ldots \\e_{I}\end{pmatrix}},{Y = \begin{pmatrix}y_{1} \\y_{2} \\\ldots \\y_{I}\end{pmatrix}}$Establishes the following residual equation from Expression (21).XW=Y+E  (22)

In this case, the tap coefficient w_(j) for obtaining the predictionvalue E[y] close to the pixel value y of the high-magnification pixelcan be obtained by minimizing the squared error

$\sum\limits_{i = 1}^{I}\;{\mathbb{e}}_{i}^{2}$

Accordingly, in the event that differentiation of the above squarederror by the tap coefficient w_(j) yields 0, this means that the tapcoefficient w_(j) which satisfies the following Expression is theoptimal value for obtaining the prediction value E[y] close to the pixelvalue y of the high-magnification pixel.

$\begin{matrix}{{{e_{1}\frac{\partial e_{1}}{\partial w_{j}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{j}}} + {\ldots\mspace{11mu} e_{I}\frac{\partial e_{I}}{\partial w_{j}}}} = {0\mspace{14mu}\left( {{j = 1},2,\ldots\mspace{11mu},J} \right)}} & (23)\end{matrix}$

Accordingly, first, differentiation of the Expression (22) with the tapcoefficient w_(j) establishes the following expression.

$\begin{matrix}{{\frac{\partial e_{i}}{\partial w_{1}} = x_{i1}},{\frac{\partial e_{i}}{\partial w_{2}} = x_{i2}},\ldots\mspace{11mu},{\frac{\partial e_{i}}{\partial w_{J}} = x_{iJ}},\left( {{i = 1},2,\ldots\mspace{11mu},I} \right)} & (24)\end{matrix}$

Expression (23) and Expression (24) yield Expression (25).

$\begin{matrix}{{{\sum\limits_{i = 1}^{I}\;{e_{i}x_{i1}}} = 0},{{\sum\limits_{i = 1}^{I}\;{e_{i}x_{i2}}} = 0},{{\ldots\mspace{14mu}{\sum\limits_{i = 1}^{I}\;{e_{i}x_{iJ}}}} = 0}} & (25)\end{matrix}$

Further, taking into consideration the relation between the student datax_(ij), tap coefficient w_(j), tutor data y_(i), and residual errore_(i) in the residual error expression of Expression (22), the followingnormal equations can be obtained from Expression (25).

$\begin{matrix}\left\{ \begin{matrix}{{{\left( {\sum\limits_{i = 1}^{I}\;{x_{i1}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{I}\;{x_{i1}x_{i2}}} \right)w_{2}} + \ldots\; + {\left( {\sum\limits_{i = 1}^{I}\;{x_{i1}x_{iJ}}} \right)w_{J}}} = \left( {\sum\limits_{i = 1}^{I}\;{x_{i1}y_{i}}} \right)} \\{{{\left( {\sum\limits_{i = 1}^{I}\;{x_{i2}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{I}\;{x_{i2}x_{i2}}} \right)w_{2}} + \ldots\; + {\left( {\sum\limits_{i = 1}^{I}\;{x_{i2}x_{iJ}}} \right)w_{J}}} = \left( {\sum\limits_{i = 1}^{I}\;{x_{i2}y_{i}}} \right)} \\\ldots \\{{{\left( {\sum\limits_{i = 1}^{I}\;{x_{iJ}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{I}\;{x_{iJ}x_{i2}}} \right)w_{2}} + \ldots\; + {\left( {\sum\limits_{i = 1}^{I}\;{x_{iJ}x_{iJ}}} \right)w_{J}}} = \left( {\sum\limits_{i = 1}^{I}\;{x_{iJ}y_{i}}} \right)}\end{matrix} \right. & (26)\end{matrix}$

Further, the normal equations of Expression (26) can be expressed asAW=v  (27)by defining the matrix (convariance matrix) A and vector v as

$\begin{matrix}{A = \begin{pmatrix}{\sum\limits_{i = 1}^{I}\;{x_{i1}x_{i1}}} & {\sum\limits_{i = 1}^{I}\;{x_{i1}x_{i2}}} & \ldots & {\sum\limits_{i = 1}^{I}\;{x_{i1}x_{iJ}}} \\{\sum\limits_{i = 1}^{I}\;{x_{i2}x_{i1}}} & {\sum\limits_{i = 1}^{I}\;{x_{i2}x_{i2}}} & \ldots & {\sum\limits_{i = 1}^{I}\;{x_{i2}x_{iJ}}} \\{\sum\limits_{i = 1}^{I}\;{x_{iJ}x_{i1}}} & {\sum\limits_{i = 1}^{I}\;{x_{iJ}x_{i2}}} & \ldots & {\sum\limits_{i = 1}^{I}\;{x_{iJ}x_{iJ}}}\end{pmatrix}} \\{v = \begin{pmatrix}{\sum\limits_{i = 1}^{I}\;{x_{i1}y_{i}}} \\{\sum\limits_{i = 1}^{I}\;{x_{i2}y_{i}}} \\\vdots \\{\sum\limits_{i = 1}^{I}\;{x_{iJ}y_{i}}}\end{pmatrix}}\end{matrix}$and defining the vector W as in Expression 20.

As many normal equations of Expression (26) can be provided as thenumber J of the tap coefficient w_(j) to be obtained, by preparing acertain number of sets of student data x_(ij) and tutor data y_(i), andaccordingly, solving Expression (27) with regard to the vector W allowsan optimal tap coefficient w_(j) to be obtained (however, the matrix Ain Expression (27) needs to be a regular matrix in order to solveExpression (27)). Note that sweeping (Gauss-Jordan elimination) or thelike, for example, may be used for solving Expression (27).

Using student data and tutor data to learn an optimal tap coefficientw_(j) beforehand, and further using the tap coefficient w_(j) to obtainthe prediction value E[y] close to the tutor value y, as describedabove, is the adaptation processing.

Note that adaptation processing is different from simple interpolationwith regard to the point that components not contained in thelow-magnification image but contained in the high-magnification imageare reproduced. That is to say, adaptation processing may seem to be thesame as simple interpolation using a so-called interpolation filter justfrom looking at Expression (20), but the tap coefficient w equivalent tothe tap coefficient of the interpolation filter can be obtained bylearning, as if it were, using the tutor data y, so components containedin the high-magnification image can be reproduced. Accordingly, it canbe said that the adaptation processing has image creating (resolutioncreating) actions, as if it were.

Next, FIG. 39 illustrates a configuration example of the scaling unit113 realized by a class classification adaptation processing circuitwhich performs class classification adaptation processing such asdescribed above. Here, the scaling unit 113 converts the all-directionalimage data into image data with a predetermined fixed magnification, andin the event that the all-directional image data is of a magnificationlower than the predetermined fixed magnification, the all-directionalimage data is converted into image data with the fixed magnificationhigher than that magnification, and conversely, in the event that themagnification of the all-directional image data is higher than thepredetermined fixed magnification, the all-directional image data isconverted into image data with the predetermined fixed magnificationlower than that magnification, with FIG. 39 illustrating a classclassification adaptation processing circuit making up the part forconverting the all-directional image data into the image data with thefixed magnification higher than that magnification.

The low-magnification image data which is image data (all-directionalimage data) output from the all-directional camera 102 _(k) is suppliedto the buffer 231, and the buffer 231 temporarily stores thelow-magnification image data supplied thereto.

A prediction tap extracting circuit 232 sequentially takes pixels ofimage data of the fixed magnification to be obtained with alater-described sum of products computing circuit 236 (hereafterreferred to as fixed magnification pixel, as appropriate) as pixels ofinterest, and further extracts from the buffer 231 pixels of thelow-magnification image data to be used for predicting the pixel ofinterest, and takes these as prediction taps.

That is, the prediction tap extracting circuit 232 reads out from thebuffer 231 several low-magnification pixels near the positioncorresponding to the fixed magnification pixel serving as the pixel ofinterest, for example, as a prediction tap.

Upon the prediction tap extracting circuit 232 obtaining a predictiontap, the prediction tap regarding that pixel of interest is supplied tothe product of sums computing circuit.

Now, control signals are supplied to the prediction tap extractingcircuit 232 from the control circuit 237, and the prediction tapextracting circuit 232 determines the low-magnification pixels toconfigure the prediction tap with, i.e., the structure of the predictiontap, according to the control signals from the control circuit 237.

That is to say, image-taking information output from the image-takingdevice 111 of the all-directional camera 102 _(k) is supplied to thecontrol circuit 237, and the control circuit 237 generates, based onzoom information, control signals for ensuring processing suitable forthe magnification which the zoom information, for example, of theimage-taking information, represents, which is supplied to theprediction tap extracting circuit 232, class tap extracting circuit 233,and coefficient storage unit 235.

Specifically, the control circuit 237 generates control signals whichinstruct configuring the prediction tap using low-magnification pixelsthat are at positions closer to the position corresponding to the pixelof interest the smaller the zoom magnification which the zoominformation indicates is, for example (control signals which instructconfiguring the prediction tap using low-magnification pixels that areat positions farther from the position corresponding to the pixel ofinterest the greater the zoom magnification is), and supplies to theprediction tap extracting unit 232.

Accordingly, the prediction tap extracting unit 232 configuresprediction taps of different structures according to the zoommagnification of the low-magnification image, according to such controlsignals.

On the other hand, the class tap extracting unit 233 extractslow-magnification pixels used for class classification for classifyingthe pixel of interest into one of several classes, from the buffer 231,and takes this as a class tap.

Now, the class tap extracting unit 233 also determines thelow-magnification pixels to configure the prediction tap with, i.e., thestructure of the class tap, according to the control signals from thecontrol circuit 237.

That is to say, the control circuit 237 supplies control signals to theclass tap extracting circuit 233 in the same way as to the predictiontap extracting circuit 232, and accordingly the class tap extractingcircuit 233 configures the class tap using low-magnification pixels thatare at positions closer to the position corresponding to the pixel ofinterest the smaller the zoom magnification which the zoom informationindicates is (configures the class tap using low-magnification pixelsthat are at positions farther from the position corresponding to thepixel of interest the greater the zoom magnification is)

Now, in order to simplify description here, let us say that theprediction tap obtained at the prediction tap extracting circuit 232 andthe class tap obtained at the class tap extracting circuit 233 have thesame tap structure, for example. However, it is needless to say that theprediction tap and class tap can have independent (separate) tapstructures.

A class tap regarding a pixel of interest obtained by the class tapextracting circuit 233 is supplied to the class classification circuit234. The class classification circuit 234 performs class classificationof the pixel of interest based on the class tap from the class tapextracting circuit 233, and outputs a class code corresponding to theconsequently obtained class.

Now, ADRC (Adaptive Dynamic Range Coding) or the like, for example, canbe used for the class classification method.

With a method using ADRC, pixel values of pixels making up a class tapare subjected to ADRC processing, and the class of the pixel of interestis determined according to the ADRC code obtained as a result thereof.

In K-bit ADRC, the maximum value MAX and minimum value MIN of multiplepixel values making up a class tap are detected for example, DR=MAX−MINis taken as a local dynamic range of a group, and the pixels making upthe class tap are re-quantized into K bits based on the dynamic rangeDR. That is to say, the smallest value MIN is subtracted from the pixelvalue of each of the pixels making up the class tap, and the subtractedvalue thereof is divided by DR/2^(K) (quantization). The pixel values ofeach of the K bits making up the class tap that are obtained asdescribed above are arrayed into a bit string according to apredetermined order, which is output as ADRC code. Accordingly, firexample, in the event that a class tap is subjected to 1-bit ADRCprocessing, the pixel value of each pixel making up the class tap havethe minimum value MIN subtracted therefrom, and then are divided by theaverage value of the maximum value MAX and minimum value MIN, so thatthe pixel value of each pixel is 1 bit (binarized). A bit string whereinthe 1-bit pixel values are arrayed in a predetermined order is output asthe ADRC code.

Now, while the level distribution (pixel value distribution) pattern ofthe pixels making up the class-tap, for example can be output from theclass classification circuit 234 without change, in this case, in theevent that the class tap is made up of N pixels and K bits are assignedto each pixel, the number of versions of class codes which the classclassification circuit 234 outputs is (2^(N))^(K), which is a greatnumber increasing exponentially according to the number of pixel bits K.

Accordingly, the class classification circuit 234 preferably performsclass classification after subjecting the amount of information of theclass tap to compressing by the above-described ADRC processing orvector quantization or the like.

Also, while the arrangement here performs class classification usinglow-magnification pixels alone, but class classification may beperformed using zoom information or exposure time or the like containedin the image-taking information, along with the low-magnification pixels(or instead of the low-magnification pixels), as class taps.

The class code output by the class classification circuit 234 isprovided to the coefficient storage unit 235 as an address.

The coefficient storage unit 235 stores tap coefficients obtained bylearning processing being performed, and outputs tap coefficients storedat addresses corresponding to the class codes output by the classclassification circuit 234 to the sum of products computing circuit 236.

Note that the coefficient storage unit 235 stores multiple sets of tapcoefficients obtained by performing learning using multiple sets oftutor data and student data, as described later. At the coefficientstorage unit 235, which set of tap coefficients to use from the multiplesets of tap coefficients is determined according to control signals fromthe control circuit 237.

That is to say, the coefficient storage unit 235 is configured ofmultiple banks, for example, with each of the banks storingcorresponding sets of the multiple sets of tap coefficients obtained byperforming learning for each range of multiple zoom magnifications. Thecontrol circuit 237 generates control signals for instructing the bankcorresponding to the zoom magnification indicated by the zoominformation, based on the zoom information in the image-takinginformation, and supplies to the coefficient storage unit 235. Thecoefficient storage unit 235 switches banks according to control signalsfrom the control circuit 237, and outputs from the tap coefficient setsstored in the selected bank, one which corresponds to the class codesupplied from the class classification circuit 234 to the sum ofproducts computing circuit 236.

The sum of products computing circuit 236 obtains the production tapoutput by the prediction tap extracting circuit 232 and the tapcoefficient output by the coefficient storage unit 235, performs thelinear prediction computation (sum-of-product computation) shown inExpression (20) using the prediction tap and the tap coefficient, andoutputs the computation results as the pixel value of the fixedmagnification pixel which is the pixel of interest.

Next, FIG. 40 illustrating a configuration example of a learning devicewhich performs learning of tap coefficients to be stored in thecoefficient storage unit 235 shown in FIG. 39.

Learning data used for learning is supplied to tutor data memory 242.Now, image data of a fixed magnification (hereafter referred to asfixed-magnification image data, as appropriate) is used as learningdata.

The tutor data memory 242 stores the fixed-magnification image datasupplied thereto as tutor data to serve as the tutor in the learning. Astudent data generating circuit 243 generates student data to serve asthe student in the learning from the tutor data stored in the tutor datamemory 242, according to the control signals supplied from a controlcircuit 251.

That is to say, the control circuit 251 has multiple zoom magnificationranges set, and the control circuit 251 selects one of the multiple zoommagnification ranges as a range of interest. The control circuit 251then determines a representative value of the zoom magnification of therange of interest (e.g., a central value of the zoom magnification ofthe range of interest), and generates control signals based on therepresentative value of the zoom magnification, which is supplied to thestudent data generating circuit 243, prediction tap extracting circuit245, and class tap extracting circuit 246.

Specifically, the control circuit 251 calculates thinning out ratio formaking the fixed-magnification image data serving as the tutor datastored in the tutor data memory 242 to be image data having the zoommagnification of the representative value, and supplies control signalsto the student data generating circuit 243 which instruct thinning outof the tutor data at that thinning out ratio.

The student data generating circuit 243 thins out the tutor data storedin the tutor data memory 242 according to the above-described controlsignals, and accordingly generates image data of the zoom magnificationof the representative value as student data. The student data issupplied from the student data generating circuit 243 to the studentdata memory 244, and is stored.

Upon student data being requested regarding the tutor data stored in thetutor data memory 242 and stored in the student data memory 244, theprediction tap extracting circuit 245 sequentially takes the tutor datastored in the tutor data memory 242 as a pixel of interest, and furtherextracts the student data to be used for predicting the pixel ofinterest from the student data memory 244, which is used as a predictiontap.

Now, the control circuit 251 generates control signals in the same wayas with the control circuit 237 in FIG. 39, based on the zoommagnification of the representative value, and supplies this to theprediction tap extracting circuit 245. The prediction tap extractioncircuit 245 then makes up a prediction tap which is the same as the caseof the prediction tap extracting circuit 232 shown in FIG. 39, based onthe control signals supplied from the control circuit 251.

Thus, the prediction tap obtained by the prediction tap extractingcircuit 245 is supplied to a normal equation adding circuit 248.

On the other hand, the class tap extracting circuit 246 extracts studentdata to be used for class classification of pixels of interest from thestudent data memory 244, and supplies to the class classificationcircuit 247 as class taps.

Now, the control circuit 251 generates control signals which are thesame as those of the control circuit 237 shown in FIG. 39 based on thezoom magnification of the representative value, and supplies this to theclass tap extracting circuit 246. The class tap extracting circuit 246configures class taps the same as the case of the class tap extractingcircuit 233 shown in FIG. 39, according to the control signals suppliedfrom the control circuit 251.

Upon receiving supply of a class tap regarding a pixel of interest fromthe class tap extracting circuit 246, the class classification circuit247 uses the class tap to perform the same class classification as theclass classification circuit 234 shown in FIG. 39, and supplies theclass code representing the class of the pixel of interest to the normalequation adding circuit 248.

The normal equation adding circuit 248 reads out the tutor data servingas the pixel of interest from the tutor data memory 242, and performsadding to the student data making up the prediction tap from theprediction tap extracting circuit 245 and the tutor data serving as thepixel of interest, for each class supplied from the class classificationcircuit 247.

That is, the normal equation adding circuit 248 performs computationequivalent to multiplication (x_(in) x_(im)) and summation (Σ) ofstudent data one with another, which are the components of the matrix Ain Expression (27), using the prediction taps (student data) for eachclass corresponding to the class code supplied from the classclassification circuit 247.

Further, the normal equation adding circuit 248 performs computationequivalent to multiplication (x_(in) y_(i)) and summation (Σ) of studentdata and tutor data, which are the components of the vector v inExpression (27), using the prediction taps (student data) and pixel ofinterest (tutor data) for each class corresponding to the class codesupplied from the class classification circuit 247, in the same way.

The normal equation adding circuit 248 performs the above-describedadding in as pixels of interest for all tutor data sorted in the tutordata memory 242, and accordingly establishes normal equations for eachclass, as shown in Equation (27).

Subsequently, a tap coefficient determining circuit 249 solves thenormal equations generated for each class at the normal equation addingcircuit 248, thereby obtaining tap coefficients for each class, which issupplied to addresses corresponding to each class in the coefficientmemory 250. Accordingly, the coefficient memory 250 stores the tapcoefficients for each of the classes supplied from the tap coefficientdetermining circuit 249.

Now, depending on the image data prepared as learning data, there may bea case wherein there is a class regarding which the necessary number ofnormal equations for obtaining tap coefficients cannot be obtained atthe normal equation adding circuit 248, but for such classes, the tapcoefficient determining circuit 249 outputs, for example, a default tapcoefficient.

Thus, upon tap coefficient sets being obtained regarding the zoommagnification of the range of interest, the control circuit 251 changesthe range of interest to another zoom magnification range, and the sameprocessing is repeated with the learning device shown in FIG. 40.Accordingly, the tap coefficient sets corresponding to each of themultiple zoom magnification ranges set by the control circuit 251 areobtained.

Next, FIG. 41 illustrates a configuration example of the coordinatescalculating unit 125 shown in FIG. 26.

The coordinates calculating unit 125 is configured of a search unit 261and image-taking point calculating unit 262, and calculates image-takingpoints to serve as the position (coordinates) for taking a desiredsubject (e.g., latitude and longitude, etc.), in response to requestfrom the control unit 118 (FIG. 26).

That is to say, subject information is supplied to the search unit 261.This subject information is input by, for example, the operating unit115 being operated by the user, and is supplied to the search unit 261via the control unit 118. Now, subject information is information fordistinguishing a desired subject, and the name of the subject (e.g.,“Tokyo Tower”, “Diet Building”, etc.), address, etc., can be used.

Upon receiving subject information, the search unit 261 makes referenceto the map data stored in the map database 124 and searches for theposition (coordinates) of the subject distinguished by the subjectinformation thereof (e.g., latitude and longitude, etc.). The searchunit 261 then supplies the position of the subject (hereafter referredto as subject point, as appropriate) to the image-taking pointcalculating unit 262.

Here, as described in FIG. 25, the three functions of the first throughthird functions are realized with the SHARN moving entity system. Of thefirst through third functions, there is no need to determine theimage-taking point for the first function, and the only time that thereis the need to determine the image-taking point is a case of realizingthe second and third functions.

In the event of realizing the second function, i.e., in the vent ofobtaining an image wherein the desired subject is viewed from a desiredviewpoint direction v, the image-taking position is determined asfollows.

That is to say, in this case, viewpoint information representing thedesired viewpoint direction v regarding the desired subject is input bythe user operating the operating unit 115.

This viewpoint information is supplied to the image-taking pointcalculating unit 262 via the control unit 118.

In the event of realizing the second function, the image-taking pointcalculating unit 262 calculates the image-taking point as a positionsuitable for taking images of the subject at the position of the subjectpoint from the viewpoint direction v which the viewpoint informationrepresents, from the search unit 261.

That is to say, the image-taking point calculating unit 262 makesreference to the map data stored in the map database 124, and therebysets the range of a place suitable for taking images of the subject as aso-called doughnut-shaped range (hereafter referred to as suitablerange, as appropriate), which is away from the subject point by adistance R_(p2) or more, but closer than a distance R_(p2).

Further, the image-taking point calculating unit 262 detects, of theintersections between line segments from the viewpoint direction vrepresented by the viewpoint information toward the subject point, androads or the like over which the moving entity 101 _(k) can move, all ofthose which are within the suitable range, as image-taking pointcandidates.

The image-taking point calculating unit 262 then selects one arbitrarypoint from the image-taking point candidates for example, and suppliesthe selected image-taking point to the control unit 118.

Here, only one position on the viewpoint direction v is taken as animage-taking point for realizing the second function, in order tosimplify description, but multiple points on the viewpoint direction vmay be taken as image-taking points. Also, examples of methods forselecting one image-taking point from the multiple image-taking pointcandidates include selecting the closest or farthest from the subjectpoint, and so forth.

Also, in the event of realizing the third function, that is, in theevent of obtaining a viewpoint-variable image wherein the desiredsubject is viewed over which changing the viewing, the image-takingpoints are obtained as follows.

That is, in the event of realizing the third function, the image-takingpoint calculating unit 262 makes reference to the map data stored in themap database 124, thereby obtaining a circle which is centered on thesubject point supplied from the search unit 261 and which has a radius Rsuitable for taking images of the subject (e.g., the average of thedistances R_(p1) and R_(p2) in FIG. 42, or the like), and obtains theintersections between that circle (perimeter) and roads and the likeover which the moving entity 101 _(k) is capable of traveling. Theimage-taking point calculating unit 262 then obtains all of theintersections as image-taking points, and supplies these to the controlunit 118.

Now, in the event of obtaining a viewpoint-variable image wherein thesubject is viewed over while changing the viewpoints, ideally, imagesare necessary wherein the subject is being taken from directions 360°around, that is to say, taken from all over.

However, there is no guarantee that all positions from which the subjectcan be viewed from directions 360° around, can be obtained asintersections between the circle having the radius R centered on thesubject point, and the roads and the like over which the moving entity101 _(k) can travel.

That is to say, in the event of taking intersections between the circlehaving the radius R centered on the subject point, and the roads and thelike over which the moving entity 101 _(k) can travel as theimage-taking points, while image-taking points which are the samedistance from the subject point can be obtained, there may be caseswherein the subject cannot be viewed from a certain direction, dependingon the state of presence of roads.

Accordingly, in the event of realizing the third function, theimage-taking point calculating unit 262 can obtain the image-takingpoints as shown in FIG. 43B, for example.

That is, in this case, the image-taking point calculating unit 262equally divides the directions 360° around the subject point as thecenter, into several directions. Here, with the embodiment in FIG. 43B,the 360° direction centered on the subject point is equally divided intoeight directions which differ by 45° each.

The image-taking point calculating unit 262 then calculates image-takingpoints in the same way as described in FIG. 42 for the eight directions,with each of the eight directions as viewpoint information. In thiscase, eight image-taking portions capable of viewing the subject fromdirections differing 45° each can be obtained. That is, in this case,the distances from the subject point to each image-taking point aregenerally not the same, but image-taking points whereby the subject canbe viewed all over from directions 360° around can be obtained.

Next, FIG. 44 illustrates the format of the all-directional image dataat the time of being stored in the image database 114 shown in FIG. 26.

In the image database 114, for example, one all-direction image data ismanaged with a header portion and data portion, as shown in FIG. 44.

The header portion has image ID (Identification) attributes information,image-taking information, and the like, and the data portion has imagedata (all-directional image data).

The image ID is a unique ID regarding the all-directional image dataplaced in that data portion, which the control unit 118 registers in theimage database 114. As for the image ID, a combination may be used of,for example: the moving entity ID of the moving entity 101 _(k) takingthe corresponding all-direction image data; the year, month, and date,on which the image was taken; which number in order of images taken onthat year, month, and date, the image is, and so forth. Let us say thatthe control unit 118, for example, stores the moving entity ID of themoving entity 101 _(k).

Attributes information represents information of various attributes inthe all-directional image data placed in the data portion. Examples ofthe attributes information include subject information fordistinguishing the subject contained in the all-directional image data,the subject point, image-taking point where the all-directional imagedata was taken, viewpoint information representing the direction ofviewing the subject from the image-taking point (viewpoint direction),and so forth. The attributes information is registered in the imagedatabase 114 by the control unit 118.

Image-taking information is information relating to settings of theimage-taking device 111, output from the image-taking device 111 asdescribed above, and is registered in the image database 114 by thescaling unit 113.

Note that the moving entity 101 _(k) is capable of receiving image datataken by the all-directional camera 102 _(k) of another moving entity101 _(k′), and image data which the moving entity 101 _(k′) has receivedfrom yet another moving entity 101 _(k″) and registered in the imagedatabase 114 thereof (the image database 114 of the moving entity 101_(k′)), so that in addition to the image data taken with its ownall-directional camera 102 _(k), image data received from other movingentities 101 _(k′) is also stored in the image database 114 of themoving entity 101 _(k), as described above.

Next, FIG. 45 illustrates a configuration example of the display imageprocessing unit 119 shown in FIG. 26.

Image data is supplied from the control unit 118 to the image memory271, and the image memory 271 stores the image data.

Note that information placed in the header portion shown in FIG. 44regarding the image data (hereafter referred to as header information asappropriate) is supplied to the image memory 271 along with the imagedata, and the image memory 271 stores the image data along with theheader information.

Also, the image memory 271 has storage capacity capable of storingmultiple sets of all-directional image data (including headerinformation).

The scaling unit 272 is configured in the same way as the scaling unit113 in FIG. 26 for example, and enlarges or reduces the image datastored in the image memory 271 according to control by the control unit118.

That is to say, in the event that the user has instructed enlarging orreducing of the image by operating the operating unit 115, the controlunit 118 controls the scaling unit 272 according to the instructions,thereby enlarging or reducing the image data stored in the image memory271.

Also, in the event that multiple sets of image data are sorted in theimage memory 271, the scaling unit 272 enlarges or reduces the size ofthe subject displayed in multiple sets of image data if necessary, sothat the subject is displayed the same size.

That is with the third function, multiple sets of image data taking thesame subject from multiple directions as described in FIG. 43 is storedin the image memory 271, in order to generate a viewpoint-variable imagewherein the desired subject is viewed over while changing theviewpoints.

In the event that the multiple sets of image data taking the samesubject from multiple directions (hereafter referred to asdifferent-direction image data) have been taken from image-taking pointsdistanced from the subject point by the same distance, as shown in FIG.34A, even if the zoom magnification at the time of image-taking at eachimage-taking point differs, this is converted into image data with thesame zoom magnification (fixed magnification) by the scaling unit 113,and accordingly, the size of the subject in the different-directionimage data is the same.

However, in the event that the multiple different-direction image datasets have been taken from image-taking points distanced from the subjectpoint by different distances as shown in FIG. 43B, converting the zoommagnification of the multiple different-direction image data sets at thescaling unit 113 does not make the size of the subject in thedifferent-direction image data sets the same size. That is to say, thesize of the subject in the different-direction image data sets dependson how far the image-taking points where the different-direction imagedata has been taken are from the subject point.

Accordingly, in the event that multiple different-direction image datasets have been stored in the image memory 271, the scaling unit 272enlarges or reduces the different-direction image data stored in theimage memory 271 so that the subject in the different-direction imagedata is the same size (scales).

Note that in this case, the enlargement ratio or reduction ratio of eachof the different-direction image data sets is determined such that thesize (scale) of the same subject in the multiple different-directionimage data sets stored in the image memory 271 is the same, based on thedistance between the image-taking points and the subject point,contained in the header thereof.

Thus, upon enlarging or reducing the image data stored in the imagememory 271, the scaling unit 272 writes the enlarged or reduced imagedata to the image memory 271, by overwriting the original image data,for example.

A corresponding pixel computing unit 273 detects corresponding pixels inthe multiple different-direction image data sets stored in the imagememory 271, in the event of realizing the third function under controlof the control unit 118.

That is, the corresponding pixel computing unit 273 makes reference tothe viewpoint information within the header information of each of thesets of the different-direction image data stored in the image memory271, thereby recognizing the order in which the viewpoint directionswhich the viewpoint information represents are clockwise orcounter-clockwise with a certain direction as the reference direction(hereafter referred to as viewpoint direction order, as appropriate).Further, the corresponding pixel computing unit 273 detects which pixelsof the subject in the i'th different-direction image data that eachpixel of the subject in the i+1'th different-direction image datacorrespond to.

Note that this corresponding pixel detection can be performed using, forexample, so-called block matching or the like.

Upon detecting the correlation of pixels regarding the multiple sets ofdifferent-direction image data stored in the image memory 271, thecorresponding pixel computing unit 273 supplies the correlation of thepixels to the image generating unit 274.

In the event of realizing the first function or second function, theimage generating unit 274 reads out the image data stored in the imagememory 271, and supplies this to the display memory 276.

Also, in the event of realizing the third function, the image generatingunit 274 uses the multiple different-direction image data sets stored inthe image memory 271 to generate the viewpoint-variable image dataviewing the subject over while changing the viewpoint in the clockwiseor counter-clockwise direction.

That is to say, the image generating unit 274 generatesviewpoint-variable image data as moving image data which would have beentaken if the subject had been taken while changing the viewpoint in theclockwise or counter-clockwise direction, by arranging thedifferent-direction image data in order of viewpoint direction as movingimage frames, while positioning in the spatial direction so as tocorrelate corresponding pixels, based on the correlation of pixelssupplied from the corresponding pixel computing unit 273.

Note that in the event that the viewpoint-variable image data isgenerated as moving image data by positioning the multipledifferent-direction image data sets stored in the image memory 271sequentially as the first frame, second frame . . . , in the viewpointdirection order, the viewpoint-variable image data may not become asmooth moving picture depending on the number of sets ofdifferent-direction image data and the density of image-taking points.

Accordingly, the image generating unit 274 is arranged to array themultiple different-direction image data sets stored in the image memory271 in an order of frames corresponding to the viewpoint direction.

That is to say, in the event of attempting to generateviewpoint-variable image data wherein the subject is circled one time(rotated 360°) in three seconds, with a frame rate of 30 frames/second,for example, the viewpoint-variable image data needs to be configured of90 frames (=30 frames/second×3 seconds), and further, the change in theviewpoint direction between adjacent frames in the viewpoint-variableimage data needs to be 4° (=360°/90 frames).

Accordingly, in this case, the image generating unit 274 positions thedifferent-direction image data closest to 4×(i−1)° in the viewpointdirection, as the i'th frame.

However, in this case, there may be frames regarding whichdifferent-direction image data is not placed. Accordingly, the imagegenerating unit 274 performs interpolation for the frames regardingwhich no different-direction image data is placed (hereafter referred toas unplaced frames, as appropriate) using frames regarding whichdifferent-direction image data is placed (hereafter referred to asplaced frames, as appropriate), by controlling an interpolation framegenerating unit 275, thereby generating image data to be placed in theunplaced frame. That is to say, the interpolation frame generating unit275 performs interpolation between a placed frame which is closest tothe unplaced frame and is ahead time-wise, and a placed frame whichfollows time-wise, thereby generating image data for the unplaced frame.Note that interpolation for this unplaced frame is performed taking intoconsideration the time-wise distance from theunplaced frame to theplaced frames ahead or following time-wise, that are used for thisinterpolation.

Now, besides performing simple interpolation using placed frame imagedata as described above, so as to generate image data for unplacedframes, the interpolation frame generating unit 275 is also capable ofgenerating unplaced frame image data by performing class classificationadaptation processing using placed frame image data, for example.

The image generating unit 274 performs necessary interpolation asdescribed above, and finally completes the viewpoint-variable image dataas moving picture data, and then supplies the viewpoint-variable imagedata to the display memory 276 where it is stored.

The image data sorted in the display memory 276 is supplied to anddisplayed at the display unit 122 (FIG. 26).

Next, processing which the moving entity 101 _(k) shown in FIG. 26performs will be described.

Note that the processing performed by the moving entity 101 _(k) can begenerally divided into processing wherein the moving entity 101 _(k)serves as a master moving entity as if it were (hereafter referred to asmaster moving entity) and other moving entities 101 _(k′) serve as slavemoving entities as if it were (hereafter referred to as slave movingentities), so as to process various requests with regard to the movingentity 101 _(k′) serving as the slave moving entity, and processingwherein the other moving entity 101 _(k′) serves as the master movingentity and processing is performed according to the requests of themaster moving entity.

Accordingly, a description will be made wherein the moving entity 101_(k) serves as the master moving entity and the moving entity 101 _(k′)serves as a slave moving entity, with regard to the processing which themoving entity 101 _(k) serving as the master moving entity performs(master moving entity processing) and processing which the moving entity101 _(k′) serving as the slave moving entity performs (slave movingentity processing).

First, processing of the master moving entity will be described withreference to the flowchart shown in FIG. 46.

First, to begin with, in step S201, the control unit 118 determineswhether or not there has been some sort of input by the user U_(k)operating the operating unit 115. In the event that determination ismade in step S201 that there has been no input, the flow returns to stepS201.

Also, in the event that there has been determined to have been someinput from the user U_(k) in step S201, the flow proceeds to step s202,where the control unit 118 determines what the input from the user U_(k)requests.

In step S202, in the event that the input from the user U_(k) isdetermined to be requesting all-directional image data, the flowproceeds to step S203, the later-described all-directional imageacquiring processing is performed, and the flow returns to step S201.

Also, in the event that determination is made in step S202 that theinput from the user U_(k) requests image data wherein a desired subjecthas been taken from a desired viewpoint direction (hereafter referred toas viewpoint-specified image data, as appropriate), the flow proceeds tostep S204, the later-described viewpoint-specified image acquiringprocessing is performed, and the flow returns to step S201.

Further, in the event that determination is made in step S202 that theinput from the user U_(k) requests viewpoint-variable image data, theflow proceeds to step S205, where the later-described viewpoint-variableimage acquiring processing is performing, and the flow returns to stepS201.

Note that the first through third functions correspond to theall-directional image acquiring processing in step S203, theviewpoint-specified image acquiring processing in step S204, and theviewpoint-variable image acquiring processing in step S205,respectively.

Next, the all-directional image acquiring processing in step S203 inFIG. 46 will be described with reference to the flowchart in FIG. 47.

In the all-directional image acquiring processing, first, to start with,in step S211, the control unit 118 determines whether or not there hasbeen input of a moving entity ID identifying another moving entity(slave moving entity) by the user U_(k) operating the operating unit115.

In step S211, in the event that determination is made that there hasbeen no input of a moving entity ID by the user U_(k), the flow proceedsto step S212, where the control unit 118 makes an image-taking requestto the image-taking device 111 of the all-directional camera 102 _(k).The image-taking device 111 takes all-directional image data in responseto the request from the control unit 118, and supplies this to thescaling unit 113 via the A/D converting unit 112.

Upon receiving supply of the all-direction image data from theall-directional camera 102 _(k), the scaling unit 113 converts (scales)the all-directional image data into the fixed-magnificationall-directional image data in step S213, the flow proceeds to step S214,and is supplied to and registered in the image database 114.

Subsequently, the flow proceeds to step S215, where the control unit 118reads out the all-directional image data registered in the imagedatabase 114 in the immediately previous step S214, and supplies this tothe image conversion unit 127, where image conversion for removingdistortion in the all-directional image data is performed.

The flow then proceeds to step S216, where the control unit 118 suppliesthe image-converted (post distortion-removal) all-directional image datato the display image processing unit 119, where the display data to bedisplayed on the display unit 122 (hereafter referred to as display dataas appropriate) is generated.

That is to say, in this case, the all-directional image data from thecontrol unit 118 is sorted in the image memory 271 at the display imageprocessing unit 119 (FIG. 45). The all-directional image data stored inthe image memory 271 is subjected to scaling adjustment at the scalingunit 272 as necessary, and is supplied to the display memory 276 via theimage generating unit 274.

The display memory 276 stores the all-directional image data suppliedvia the image generating unit 274 as display data, and supplies this tothe display unit 122.

Accordingly, in step S216, the all-directional image data is displayedon the display unit 122, and the all-directional image data acquiringprocessing ends.

Accordingly, in this case, the all-directional image data (still images)obtained on site by the master moving entity is displayed on the displayunit 122.

On the other hand, in step S211, in the event that determination is madethat there has been input of a moving entity ID by the user U_(k), theflow proceeds to step S217, and the control unit 118 controls thetransmission unit 116 so as to transmit request signals requesting imagedata along with the moving entity ID which the user U_(k) has inputincluded therein.

In this case, as described later, the moving entity (slave movingentity) distinguished by the moving entity ID included in the requestsignal takes the all-directional image data on site, and transmits thisto the moving entity 101 _(k) which is the master moving entity, so instep S218, the reception unit 117 awaits and receives transmission ofthe all-directional image data. Note that the slave moving entitytransmits the all-directional image data with the same format as thestorage format of the image database 114 shown in FIG. 44.

Subsequently, the flow proceeds to step S214, where the all-directionalimage data received at the reception unit 117 is supplied to andregistered in the image database 114. Hereafter, the same processing asdescribed above is performed in steps S215 and S216, and theall-directional image data acquiring processing ends.

Accordingly, in this case, the all-directional image data (still images)taken on site by the slave moving entity identified by the moving entityID input by the user U_(k) is displayed on the display unit 122.

Next, viewpoint-specified image obtaining processing in step S204 ofFIG. 46 will be described with reference to the flowchart in FIG. 48.

First, to begin with, in step S211, the control unit 118 controls thedisplay unit 122 so as to display a message requesting subjectinformation and viewpoint information input, the flow proceeds to stepS222, and the control unit 118 determines whether the user U_(k) hasinput the subject information and viewpoint information by operating theoperating unit 115.

In step S222, in the event that determination is made that the subjectinformation and viewpoint information has not been input, the flowreturns to step S211, and the user is requested to input the subjectinformation and viewpoint information.

Also, in step S222, in the vent that determination has been made thatthere has been input of subject information and viewpoint information,the flow proceeds to step S223, where the control unit 118 supplies thesubject information and viewpoint information to the coordinatescomputing unit 125, so as to calculate the image-taking point. That isto say, the coordinates computing unit 125 obtains one image-takingpoint in the way described with FIG. 42, and supplies this to thecontrol unit 118.

The control unit 118 stores the image-taking point from the coordinatescomputing unit 125 to the storage unit 121, the flow proceeds to stepS224, where the image database 114 is referred to, thereby determiningwhether all-directional image data including the subject distinguishedby the subject information input by the user U_(k) has been alreadyregistered for the image-taking point obtained in step S223. thisdetermination can be made by referring to the information (headerinformation) of the header portion (FIG. 44) in the image database 114.

In step S224, in the event that determination is made thatall-directional image data including the subject distinguished by thesubject information input by the user U_(k) has been already registeredin the image database 114 for the image-taking point obtained in stepS223, i.e., in the event that all-directional image data viewing thesubject identified by the subject information which the user U_(k) hasinput from the viewpoint direction which the viewpoint information alsoinput by the user U_(k) represents has been taken by itself in the pastor has been received from another moving entity has already beenregistered in the image database 114, the flow proceeds to step S225,where the control unit 118 reads out the all-directional image data fromthe image database 114, and the flow proceeds to step S232.

Also, in step S224, in the event that determination is made thatall-directional image data including the subject distinguished by thesubject information input by the user U_(k) has not been registered inthe image database 114 for the image-taking point obtained in step S223,the flow proceeds to step S226, and the comparing unit 120 determineswhether or not the current position matches the image-taking pointstored in the storage unit 121.

That is, in step S226, determination is made by the comparing unit 120regarding whether or not the current location detected by the positiondetecting unit 123 matches the image-taking point stored in the storageunit 121 in step S223.

In the event that determination is made in step S226 that the currentposition matches the image-taking position stored in the storage unit121, i.e., in the event that the master moving entity itself is situatedat the image-taking position, the flow proceeds to step S227, and thecontrol unit 118 makes an image-taking request to the image-takingdevice 111 of the all-directional camera 102 _(k). The image-takingdevice 111 takes the all-directional image in response to the requestfrom the control unit 118, and supplies this to the scaling unit 113 viathe A/D converting unit 112.

Upon receiving supply of the all-directional image data, the scalingunit 113 converts (scales) the all-directional image data intoall-directional image data of a fixed magnification in step S228, theflow proceeds to step S229, and this is supplied to and registered inthe image database 114. The control unit 118 reads out theall-directional image data registered in the image database 114 in stepS229, and the flow proceeds to step S232.

On the other hand, in step S226, in the event that determination is madethat the current location does not match the image-taking point storedin the storage unit 121, the flow proceeds to step S230, where thecontrol unit 118 controls the transmission unit 116 so as to transmit arequest signal requesting image data to the image-taking points obtainedin step S233, with the image-taking point obtained in step S233 includedtherein.

In this case, as described later, a moving entity (slave moving entity)located at the image-taking point contained in the request signal takesthe all-directional image data at the current location (i.e.,image-taking point) or the like and transmits this to the moving entity101 _(k) which is the master moving entity, so in step S231, thereception unit 117 awaits and receives transmission of theall-directional image data.

In this case, moving entity processing shown in FIG. 50 described lateris performed for multiple moving entities, so it is conceivable that thesame all-directional image data taken from the image-taking pointcontained in the request signal may be transmitted, and in the eventthat the same all-directional image data is transmitted from multiplemoving entities in this way, the reception unit 117 selects one of thesame multiple all-directional image data, for example. This is the samefor step S250 in FIG. 49 described later, as well.

Subsequently, the flow proceeds to step S229, where the all-directionalimage data received by the reception unit 117 is supplied to andregistered in the image database 114. the control unit 118 reads out theall-directional image data registered in the image database 114 in stepS229, and the flow proceeds to step S232.

In step S232, the control unit 118 supplies the all-directional imagedata read out from the image database 114, i.e., the viewpoint-specifiedimage data which is all-directional image data viewing the subjectdistinguished by the subject information input by the user U_(k) fromthe viewpoint direction which the viewpoint information also input bythe user U_(k) represents, to the cropping unit 126 along with theheader information thereof, so that the portion of theviewpoint-specified image data containing the subject distinguished bythe subject information input by the user U_(k) is cropped out.

That is to say, the portion of the all-directional image data which isthe viewpoint-specified image data containing the subject can berecognized from the subject point and image-taking point in the headerinformation. The cropping unit 126 crops out a region of theall-directional image data which is the viewpoint-specified image datacontaining the subject, of a region surrounded by arcs F_(C1) and F_(C2)of circles centered on the center point C1 and straight lines L_(C1) andL_(C2) passing through the center point C1, and the image data of theregion that has been cropped out (hereafter referred to as cropped imagedata, as appropriate) is supplied to the control unit 118.

Upon receiving the cropped image data, the control unit 118 proceeds tostep S233 and supplies the cropped image data to the image convertingunit 127, where image conversion is performed to remove the distortion.

The flow then proceeds to step S234, where the control unit 118 suppliesthe converted (post distortion removal) cropped image data to thedisplay image processing unit 119, and generates display data.

That is to say, in this case, at the display image processing unit 119(FIG. 45), the cropped image data from the control unit 118 is stored inthe image memory 271. The cropped image data stored in the image memory271 is subjected to scale adjustment in the scaling unit 272 asnecessary, and is supplied to the display memory 276 via the imagegenerating unit 274.

The display memory 276 stores the cropped image data supplied via theimage generating unit 274 as display data, and supplies this to thedisplay unit 122. Accordingly, in step S234, the display unit 122displays the cropped image data, and the viewpoint-specified imageacquiring processing ends.

Accordingly, in this case, the display unit 122 displays images (stillimages) of the subject desired by the user, taken from the viewpointdirection desired by the user.

Next, the viewpoint-variable image acquiring processing in step S205 ofFIG. 46 will be described with reference to the flowchart in FIG. 49.

First, to start with, in step S241, the control unit 118 controls thedisplay unit 122 so as to display a message requesting input of thesubject information, an the flow proceeds to step S242, where thecontrol unit 118 determines whether or not the user U_(k) has inputsubject information by operating the operating unit 115.

In step S242, in the event that determination is made that subjectinformation has not been input, the flow returns to step S241, and inputof the subject information is requested for the user again.

Also, in step S242, in the event that determination is made that therehas been input of subject information, the flow proceeds to step S243,where the control unit 118 supplies the subject information to thecoordinates computing unit 125 so as to calculate the image-takingpoint. That is to say, in this case, the coordinates computing unit 125obtains the multiple image-taking points for taking the subjectdistinguished by the subject information as described in FIG. 43, andsupplies these to the control unit 118.

The control unit 118 stores the multiple image-taking points from thecoordinates computing unit 125 to the storage unit 121 where they arestored, the flow proceeds to step S244, and reference is made to theimage database 114, whereby determination is made regarding whether ornot there is an image-taking point of the subject distinguished by thesubject information, of the multiple-image taking points stored in thestorage unit 121, already registered in the image database 114.

In the event that determination is made in step S244 that there is animage-taking point of the subject distinguished by the subjectinformation, of the multiple-image taking points stored in the storageunit 121, already registered in the image database 114, i.e., in theevent that there is all-directional image data taken of the subjectdistinguished by the subject information from an image-taking pointstored in the storage unit, already registered in the image database114, the flow proceeds to step S245, where the control unit 118 readsout the all-directional image data from the image database 114, and theflow proceeds to step S246.

In step S246, the control unit 118 supplies the all-directional imagedata read out from the image database 114 to the cropping unit 126 alongwith the header information thereof, so that the portion of theall-directional image data containing the subject distinguished by thesubject information input by the user U_(k) is cropped out.

That is to say, as described above, the portion of the all-directionalimage data containing the subject can be recognizes from the subjectpoint and image-taking point in the header information, so the croppingunit 126 crops out the portion of image data of the all-directionalimage data containing the subject, and the image data of the region thathas been consequently cropped out (including header information) issupplied to the control unit 118.

The control unit 118 temporarily stores the image data supplied in thisway from the cropping unit 126 in internal memory (not shown), and theflow proceeds to step S247.

In step S247, the control unit 118 deletes those of the multipleimage-taking points stored in the storage unit 121 which match theimage-taking point of the all-directional image data read out from theimage database 114 in step S245, and the flow proceeds to step S248.

Accordingly, the storage unit 121 stores only image-taking points in theviewpoint directions in which all-directional image data has not beenobtained regarding the subject distinguished by the subject informationinput by the user U_(k).

Note that in the event that there are two or more of the multipleimage-taking points stored in the storage unit 121 which have beenalready registered in the image database 114 as image-taking points ofthe subject distinguished by the subject information, the processing inthe above-described steps S245 through S247 is performed for the two ormore image-taking points already registered.

On the other hand, in the event that determination is made in step S244that here are none of the multiple image-taking points stored in thestorage unit 121 which have been already registered in the imagedatabase 114 as image-taking points of the subject distinguished by thesubject information, i.e., in the event that there is no all-directionalimage data containing the subject distinguished by the subjectinformation, taken from the image-taking points stored in the storageunit 121, registered in the image database 114, the flow proceeds tostep S248, where the control unit 118 controls the transmission unit 116so as to transmit a request signal requesting image data along with theimage-taking points stored in the storage unit 121, and the flowproceeds to step S249.

In step S249, the reception unit 117 determines whether all-directionalimage data which has been taken at one of the image-taking pointscontained in the request signal has been transmitted from a movingentity (slave moving entity) in response to the request signaltransmitted in step S248, and in the event that determination is madethat this has been transmitted, the flow proceeds to step S250, wherethe reception unit 117 receives the all-directional image data. Thereception unit 117 then proceeds to step S254, supplies theall-directional image data received in step S250 to the image database114 where it is registered, and the flow proceeds to step S255.

On the other hand, in the event that determination is made in step S249that no all-directional image data has been transmitted from a movingentity (slave moving entity), the flow proceeds to step S251, where thecomparing unit 120 determines whether or not the current positionmatches any of the image-taking points stored in the storage unit 121,in the same way as step S226 in FIG. 48.

In step S251, in the event that determination is made that the currentlocation does not match any of the image-taking points stored in thestorage unit 121, the flow skips the step S252 through S256 and proceedsto step S257.

Also, in the event that determination is made in step S251 that thecurrent position matches one of the image-taking positions stored in thestorage unit 121, i.e., in the event that the master moving entityitself is situated at an image-taking position, the flow proceeds tostep S252, and the control unit 118 makes an image-taking request to theimage-taking device 111 of the all-directional camera 102 _(k). Theimage-taking device 111 takes the all-directional image in response tothe request from the control unit 118, and supplies this to the scalingunit 113 via the A/D converting unit 112.

Upon receiving supply of the all-directional image data, the scalingunit 113 converts (scales) the all-directional image data intoall-directional image data of a fixed magnification in step S253, theflow proceeds to step S254, and this is supplied to and registered. inthe image database 114, and the flow proceeds to step S255.

In step S255, the control unit 118 deletes those of the multipleimage-taking points stored in the storage unit 121 which match theimage-taking points of the all-directional image data stored in theimage database 114 in step S254, and proceeds to step S256.

In step S256, the control unit 118 reads out the all-directional imagedata registered in the image database 114 in step S254, and suppliesthis to the cropping unit 126 along with the header information thereof,in the same way as with step S246. Accordingly, the portion of theall-directional image data containing the subject distinguished by thesubject information input by the user U_(k) is cropped out by thecropping unit 126, based on the header information from the control unit118. The cropping unit 126 then supplies the image data of the regionthat has been consequently cropped out to the control unit 118, and thecontrol unit 118 temporarily stores the image data supplied from thecropping unit 126 in internal memory, and the flow proceeds to stepS257.

In step S257, the control unit 118 determines whether or not an imagetaking point is stored in the storage unit, and whether or not apredetermined amount of time has elapsed following transmitting therequest signal in step S248. In step S257, in the event thatdetermination is made that an image-taking point is registered in thestorage unit 121, and that the predetermined amount of time has notelapsed since transmitting the request signal, the flow returns to stepS249, and the same processing is repeated.

Also, in step S257, in the event that determination is made that thereis no image-taking point registered in the storage unit 121, or that thepredetermined amount of time has elapsed since transmitting the requestsignal, that is to say, in the event that the all-directional image datataken of the subject distinguished by the subject information which theuser U_(k) has input, from all of the multiple image-taking pointsobtained in step S243, or such all-directional image data has not beenobtained following elapsing of the predetermined amount of timefollowing transmitting the request signal, the flow proceeds to stepS258, where the control unit 118 clears the stored contents of thestorage unit 121, and the flow proceeds to step S259.

In step S259, the control unit 118 supplies all of the cropped imagedata stored in the internal memory to the image converting unit 127, sothat image conversion to remove the distortion thereof is performed.

The flow then proceeds to step S260, where the control unit 118 suppliesthe cropped image data following image conversion (after distortionremoval) to the display image processing unit 119, so as to generatedisplay data.

That is to say, in this case, with the display image processing unit 119(FIG. 45), the cropped image data supplied from the control unit 118 isstored at the image memory 271 as the above-describeddifferent-direction image data. The different-direction image datastored in the image memory 271 has the scaling thereof adjusted at thescaling unit 272 as necessary.

Then, the corresponding pixel computing unit 273 detects the correlationof pixels with regard to the different-direction image data stored inthe image memory 271 in the same way as describe above, and supplies tothe image generating unit 274.

The image generating unit 274 generates viewpoint-variable image data,i.e., data changing the viewpoint in the clockwise or counter-clockwisedirection, using the different-direction image data stored in the imagememory 271, while making reference to the correlation of the pixels fromthe corresponding pixel computing unit 273, thereby generating imagedata taking the subject distinguished by the subject information whichthe user U_(k) has input, and supplies this as display data to thedisplay memory 276.

The display memory 276 stores the viewpoint-variable image data suppliedvia the image generating unit 274 as display data, and supplies this tothe display unit 122. Accordingly, in step S260 the viewpoint-variableimage data which is moving picture data is displayed on the display unit122, and the viewpoint-variable image acquiring processing ends.

Accordingly, in this case, the display unit 122 displays a movingpicture which would have been taken if the subject desired by the userhad been taken while circling the perimeter thereof.

Note that in the image generating unit 274 of the display imageprocessing unit 119 (FIG. 45), in the event that viewpoint-variableimage data is generated using the different-direction image data storedin the image memory 271, image data to be placed in unplaced frames isgenerated at the interpolation frame generating unit 275 for the framesmaking up the moving picture serving as the viewpoint-variable imagedata where the different-direction image data has not been placed(unplaced frames), as described with FIG. 45.

Accordingly, the viewpoint-variable image is smoother (has higherresolution in the time direction) the fewer the number of unplacedframes is, meaning that the more sets of different-direction image datacan be collected, the more the image quality of the viewpoint-variableimage is improved.

Next, the processing of the slave moving entity will be described withreference to FIG. 50.

With the processing of the slave moving entity, in step S271, thereception unit 117 determines whether or not a request signal has beentransmitted from the master moving entity which is another movingentity, and in the event that determination is made that there has beenno transmission, the flow returns to step S271.

In the event that determination is made in step S271 that a requestsignal has been transmitted, the reception unit 117 receives the requestsignal which is supplied to the control unit 118, and the flow proceedsto step S272.

In step S272, the control unit 118 determines whether or not a movingentity Id is contained in the request signal from the reception unit117, and in the event that determination is made that this is contained,the flow proceeds to step S273.

In step S273, the control unit 118 determines whether or not the movingentity ID contained in the request signal matches its own moving entityID. Now, the control unit 118 stores the own moving entity ID ininternal memory, for example.

In step S273, in the event that determination is made that the movingentity ID contained in the request signal does not match the own movingentity ID, the flow returns to step S271, and subsequently, the sameprocessing is repeated.

Also, in step S273, in the event that determination is made that themoving entity ID contained in the request signal matches the own movingentity ID, the flow proceeds to step S274, where the control unit 118performs a request for image-taking to the image-taking device 111. Theimage-taking device 111 takes the all-directional image data in responseto the request from the control unit 118, and supplies this to thescaling unit 113 via the A/D converting unit 112.

Upon receiving supply of the all-directional image data, the scalingunit 113 converts (scales) the all-directional image data intoall-directional image data of a fixed magnification in step S275, theflow proceeds to step S276, and this is supplied to and registered inthe image database 114, and the flow proceeds to step S277.

In step S277, the control unit 118 reads out the all-directional imagedata registered in the image database 114 in step S276 along with theheader information, supplies this to the transmission unit 116 so as tocause the transmission unit 116 transmit this to the master movingentity transmitting the request signals. The flow then returns to stepS271, and the same processing is repeated thereafter. Note that therequest signal contains the moving entity ID of the moving entity whichhas transmitted the request signal, and in step S277 (as well as thelater-described steps S282 and S288), the all-directional image data istransmitted with the moving entity ID as the destination thereof.

On the other hand, in step S272, in the event that determination is madethat there is no moving entity ID contained in the request signal, theflow proceeds to step S278, where the control unit 118 makesdetermination whether or not one or more image-taking points arecontained in the request signal. In the event that determination is madein step S278 that there are not image-taking points contained in therequest signals, the flow returns to step S271, and the same processingis repeated thereafter.

On the other hand, in step S278, in the event that determination is madethat one or more image-taking points are contained in the requestsignal, the flow proceeds to step S279, where the control unit 118supplies the one or more image-taking points contained in the requestsignal to the storage unit 121 for storage.

Now, in the event that the viewpoint-specified image acquiringprocessing shown in FIG. 48 is to be performed at the master movingentity, only one image-taking point is included in the request signal,while in the event that the viewpoint-variable image acquiringprocessing shown in FIG. 49 is to be performed, one or more image-takingpoints are included in the request signal.

Subsequently, the flow proceeds to step S280, where the control unit 118makes reference to the image database 114 to determine whether or notthere is an image-taking point in the one or more image-taking pointsstored in the storage unit 121 which matches an image-taking point forall-directional image data stored in the image database 114.

In step S280, in the event that determination is made that there is animage-taking point in the one or more image-taking points stored in thestorage unit 121 which matches an image-taking point for all-directionalimage data stored in the image database 114, i.e., in the event thatall-directional image data taken from an image-taking point stored inthe storage unit 121 has already been registered in the image database114, the flow proceeds to step S282, the control unit 118 reads out theall-directional image data thereof from the image database 114, and theflow proceeds to step S282.

In step S282, the control unit 118 supplies the all-directional imagedata read out from the image database 114 to the transmission unit 116with the header information thereof so as to cause the transmission unit116 to transmit this to the master moving entity which transmitted therequest signal. The flow then proceeds to step S283, where the controlunit 118 deletes those of the one or more image-taking points stored inthe storage unit 121 which match the image-taking points of theall-directional image data read out from the image database 114 in stepS281, and the flow proceeds to step S284.

Note that in the event that there are multiple image-taking pointsstored in the storage unit 121 which match the image-taking points ofthe all-directional image data stored in the image database 114, theprocessing of the above-described steps S281 through S283 is performedregarding the multiple image-taking points.

On the other hand, in the event that determination is made in step S280that there are no image-taking points stored in the storage unit 121which match the image-taking points of the all-directional image datastored in the image database 114, i.e., in the event thatall-directional image data requested by the master moving entity is notregistered in the image database 114 of the slave moving entity, theflow proceeds to step S284, where the comparing unit 120 determineswhether the current position matches one of the image-taking pointsstored in the storage unit 121, as described in step S226 in FIG. 48.

In the event that determination is made in step S284 that the currentposition matches one of the image-taking positions stored in the storageunit 121, i.e., in the event that the slave moving entity is situated atone of the image-taking positions contained in the request signal, theflow proceeds to step S285, and the control unit 118 makes animage-taking request to the image-taking device 111. The image-takingdevice 111 takes the all-directional image in response to the requestfrom the control unit 118, and supplies this to the scaling unit 113 viathe A/D converting unit 112.

Upon receiving supply of the all-directional image data, the scalingunit 113 converts (scales) the all-directional image data intoall-directional image data of a fixed magnification in step S286, theflow proceeds to step S287, this is supplied to and registered in theimage database 114, the flow proceeds to step S288.

In step S288, the control unit 118 reads out the all-directional imagedata registered in the image database 114 in step S287 along with theheader information thereof, supplies this to the transmission unit 116so as to cause the transmission unit 116 to transmit this to the mastermoving entity which has transmitted the request signals.

Subsequently, the flow proceeds to step S289, where the control unit 118deletes those of the one or more image-taking points stored in thestorage unit 121 which match the image-taking points of theall-directional image data stored in the image database 114 in stepS287, and the flow returns to step S284.

On the other hand, in step S284, in the event that determination is madethat the current location does not match any of the image-taking pointsstored in the storage unit 121, the flow proceeds to step S290, wherethe control unit 118 determines whether there are not image-takingpoints stored in the storage unit 121, and whether or not apredetermined amount of time has elapsed following receiving the requestsignal from the master moving entity. In step S290, in the event thatdetermination is made that an image-taking point is stored in thestorage unit 121, and that the predetermined amount of time has notelapsed since receiving the request signal, the flow returns to stepS284, and the same processing is repeated.

Also, in step S290, in the event that determination is made that thereis no image-taking point stored in the storage unit 121, or that thepredetermined amount of time has elapsed since transmitting the requestsignal, i.e., that all of the all-directional image data taken from theone or more image-taking points contained in the request signal has beentransmitted to the master moving entity, or that part or all of theall-directional image data taken from each of the one or moreimage-taking points contained in the request signal is not obtainedafter a predetermined amount of time has elapsed following reception ofthe request signal, the flow proceeds to step S291, where the controlunit 118 clears the stored contents of the storage unit 121. The flowthen returns to step S271, where the same processing is repeatedthereafter.

As described above, the slave moving entity takes the all-directionalimage data with the current location as the image-taking point andtransmits this to the master moving entity, in the event that its ownmoving entity ID is contained in the request signal. Also, in the eventthat one or more image-taking points are contained in the request signaland all-directional image data of the image-taking points is stored inthe image database 114, the slave moving entity transmits theall-directional image data to the master moving entity, and in the eventthat this is not stored and the own current location matches animage-taking point, takes the all-directional image data at the currentlocation which is the image-taking point and transmits this to themaster moving entity.

Accordingly, in a case of including the moving entity ID in the requestsignal, the master moving entity can obtain all-directional image datataken at the current location of the slave moving entity, from the slavemoving entity of that moving entity ID (first function).

Also, in the event that the master moving entity has transmitted therequest signal including one image-taking point, all-directional imagedata which a slave moving entity has taken at the current locationthereof can be obtained from a slave moving entity of which the currentlocation is the image-taking point. Or, the master moving entity canobtain from a slave moving entity having all-directional image data ofan image-taking point contained in the request signal, theall-directional image data. As a result, the master moving entity canobtain all-directional image data containing a desired subject viewedfrom a desired viewpoint direction (second function).

Also, in the event that the master moving entity has transmitted therequest signal including multiple image-taking points, all-directionalimage data which multiple slave moving entities have taken at thecurrent locations thereof can be obtained from multiple slave movingentities of which the current locations are the image-taking points. Or,the master moving entity can obtain from slave moving entities havingall-directional image data of the image-taking points contained in therequest signal, the all-directional image data. As a result, the mastermoving entity can obtain all-directional image data containing a desiredsubject viewed from multiple viewpoint directions, and further, canobtain viewpoint-variable images which is a moving picture wherein thedesired subject is viewed over from therearound, from the multipleall-directional image data sets.

That is, with the all-directional cameras 102 ₁, 102 ₂, . . . , 102_(K), of the moving entities 101 ₁, 101 ₂, . . . , 101 _(K),individually obtaining a viewpoint-variable image necessitates movingaround the subject and taking the subject from multiple directions,thereby obtaining multiple sets of all-directional image data, so aviewpoint-variable image cannot be immediately obtained.

Accordingly, with the SHARN moving entity system, multiple movingentities collaborate (cooperate) with each other so as to obtainall-directional image data taken from each of multiple image-takingpoints. That is, the master moving entity requests the all-directionalimage data taken from each of multiple image-taking points from theslave moving entities, and slave moving entities having theall-directional image data or slave moving entities capable of takingthe all-directional image data provide the all-directional image data tothe master moving entity.

In this way, the master moving entity obtains the necessaryall-directional image data through the cooperation of slave movingentities having the necessary all-directional image data or slave movingentities capable of taking the necessary all-directional image data, andthereby is capable of obtaining a viewpoint-variable image which couldnot be obtained with it own camera alone, that is to say, to obtain amoving picture which is not simply of a subject but has the added valueof the subject appearing as if it had been taken while moving around thesubject.

Accordingly, generating (taking) viewpoint-variable images can berealized at the master moving entity, which was difficult with theall-directional camera thereof alone, the reason thereof being that themaster moving entity and one or more slave moving entities act as ifthey were a single apparatus as indicated by being surrounded by thedotted line in FIG. 51, such that the slave moving entities provide theall-directional image data of the image-taking points which the mastermoving entity requests, and the master moving entity generatesviewpoint-variable image data using the all-directional image dataprovided from the slave moving entities in this way, wherebyconsequently the all-directional cameras of the multiple moving entitiesshare the task of taking (generating) viewpoint-variable image datathrough mutual collaboration.

Conversely, in the event that the slave moving entities simply providedarbitrarily taken all-directional image data to the master movingentity, this is simply sharing the task of collecting all-directionalimage data among the slave moving entities, so images necessary forgenerating a viewpoint-variable image cannot be obtained at the mastermoving entity, and accordingly, the high-functionality of generating theviewpoint-variable image cannot be realized.

Note that with the SHARN moving entity system, in the event of each ofthe all-directional cameras of multiple moving entities sharing theprocessing of taking (generating) viewpoint-variable image data throughmutual collaboration, the viewpoint-variable image data can beimmediately obtained, so taking note of this point, it can be said thatreal-time collaboration sharing is being performed, as if it were.

On the other hand, with the SHARN moving entity system, due to thereal-time collaboration sharing being carried out such as describedabove, all-directional image data taken at multiple image-taking pointsnecessary for generating a variable-variable image of a predeterminedsubject is accumulated at the image database 114 of the moving entitywhich is the master moving entity. Consequently, in the event that therequests for generating viewpoint-variable images is great for apredetermined subject, viewpoint-variable images with hightime-resolution can be generated according to the number of times ofrequest.

That is, for example, in the event of generating a viewpoint-variableimage which circles around the subject in 3 seconds with a frame rate of30 frames/second, ideally, 90 images taking the subject from viewpointswhich differ by 4° each (different direction images), and upon the 90different direction images being collected, a viewpoint-variable imagewith the highest time-resolution can be obtained under the conditions ofcircling around the subject in 3 seconds with a frame rate of 30frames/second.

In this case, there is the need to obtain all-directional images takenat 90 image-taking points (90 all-directional images which aredifferent-direction images), and assuming there is no all-directionalimage data in any of the image databases 114 of all of the movingentities in order to simplify description, even if one moving entity Atransmits request signals including the 90 image-taking points for thepredetermined subject in the viewpoint-variable image acquiringprocessing (FIG. 49), there is no guarantee that there will be movingentities at each of the 90 locations, and accordingly, generally, notall of the all-directional images taken at the 90 image-taking pointswill be collected.

However, saying that the all-directional images of 30 image-takingpoints, which is ⅓ of the 90 image-taking points, have been collectedand registered in the image database 114 of the moving entity A forexample, by the viewpoint-variable image acquiring processing by themoving entity A, in the event that another moving entity B subsequentlytransmits a request signal including the 90 image-taking points for thepredetermined subject, in the viewpoint-variable image acquiringprocessing (FIG. 49), this moving entity B will be able to obtain atleast the 30 all-directional images registered in the image database 114of the moving entity A. Further, in this case, there is high probabilitythat other moving entities are situated at some of the remaining 60image-taking points, so the moving entity B can obtain theall-directional images taken from some of the remaining 60 image-takingportions, from such moving entities.

Accordingly, in the event that there is a request for generatingviewpoint-variable images from multiple moving entities for apredetermined subject, all-directional images of a great number ofimage-taking points are obtained according to the number of times ofrequest, and consequently, viewpoint-variable images with hightime-resolution are generated.

In this way, with regard to subjects regarding which there are manyrequests for generating viewpoint-variable images, viewpoint-variableimages having the added value of high time-resolution are generatedaccording to the number of times of request, which is due to themultiple all-directional cameras of the multiple moving entitiesperforming the processing of taking (generating) viewpoint-variableimage data through mutual collaboration. However, taking note ofgenerating viewpoint-variable images with the added value of hightime-resolution, the high time-resolution is realized by requests forgenerating viewpoint-variable images of the predetermined subject beingmade many times, so passing of a certain amount of time is necessary,and accordingly, it can be said the long-term collaboration sharing iscarried out with regard to the above-described real-time collaborationsharing.

Note that while in the above-described case, image-taking is performedwith an all-directional camera 102 _(k) from the moving entity 101 _(k),but an arrangement may be made wherein the moving entity 101 _(k) takespictures with a normal camera.

Also, ships or aircraft or the like may be employed as the moving entity101 _(k), besides automobiles.

Further, while in the above-described case, viewpoint-variable imageswhich appear as if the image was taken while circling the subject aregenerated as the added-value images with the SHARN moving entity system,but other examples of added-value images include, for example,generating a three-dimensional image using so-called triangulationsurveying.

Next, the above-described series of processing may be carried out byhardware, or may be carried out by software. In the event of carryingout the series of processing with software, a program making up thesoftware is installed in an general-purpose computer or the like.

Accordingly, FIG. 52 illustrates a configuration example of anembodiment of a computer to which the program for executing theabove-described series of processing is installed.

The program can be stored beforehand in a hard disk 2105 or ROM 2103serving as recording media built into the computer.

Or, the program may be temporarily or permanently stored (recorded) in aremovable recording medium 2111 such as a flexible disk, CD-ROM (CompactDisk Read Only Memory), MO (Magneto optical) disk, DVD (DigitalVersatile Disk), magnetic disk, semiconductor memory, or the like. Suchremovable recording media 2111 can be provided as so-called packagedsoftware.

Note that besides being installed into the computer from such aremovable recording medium 2111 as described above, the program may betransmitted from a download site to the computer wirelessly via asatellite for digital satellite broadcast, or transmitted through cableto the computer via a network such as a LAN (Local Area Network), theInternet, or the like, and installed to the internal hard disk 2105 ofthe computer by receiving the program being transmitted in such a waywith a communication unit at the computer.

The computer has a built-in CPU (Central Processing Unit) 2102. Aninput/output interface 2110 is connected to the CPU 2102 via the bus2101, and upon commands being input by a user operating or the like aninput unit 2107 comprises a keyboard, mouse, microphone, etc., theprograms stored in ROM (Read Only Memory) 2103 is executed accordingly.Or, the CPU 2102 loads to RAM (Random Access Memory) 2104 and executesthe program stored in the hard disk 2105, the program transmitted fromsatellite or network and received by the communication unit 2108 andinstalled in the hard disk 2105, or the program installed in the harddisk 2105 by being read out from a removable recording medium 2111mounted to the drive 2109. Accordingly, the CPU 2102 carried outprocessing following the above-described flowcharts, or processingperformed by the above-described block diagram configurations. Theprocessing results of the CPU 2102 are output from an output unit 2106configured of a LCD (Liquid crystal Display) and speaker or the like,transmitted from the communication unit 2108, recorded in the hard disk2105, or the like, via the input/output interface 2110, for example, asnecessary.

Now, with the present specification, the processing steps describing theprogram for causing the computer to carry out the various types ofprocessing do not necessarily need to be processed in time-sequencefollowing the order given in the flowcharts, and may include parallel orindividually executed processing (for example, parallel processing orprocessing by object).

Also, the program may be processed by a single computer, or may beprocessed in a scattered manner among multiple computers. Further, theprogram may be transferred to and executed on a remote computer.

INDUSTRIAL APPLICABILITY

According to the information processing system, first informationprocessing method, first program and first recording medium according tothe present invention, multiple acting means or multiple detecting meansshare processing through collaboration, so the functionality of aninformation processing system is raised.

According to the information processing device, second informationprocessing method, second program and second recording medium accordingto the present invention, acting means or detecting means shareprocessing through collaboration with another information processingdevice, so the functionality of an information processing device israised.

1. An information processing system comprising: a plurality of actuatorsconfigured to act corresponding to operations by a plurality of users; aplurality of detectors configured to detect the real world where saidplurality of actuators act; a communication unit configured to exchangeinformation between said plurality of actuators and said plurality ofdetectors; a storage unit configured to store results from acting bysaid plurality of actuators or results from detecting by said pluralityof detectors; a comparing unit for comparing: a present result fromacting by one or more of said plurality of actuators with a past resultfrom acting by one or more of said plurality of actuators and stored insaid storage unit; and a present result from detecting by one or more ofsaid plurality of detectors with a past result from detecting by one ormore of said plurality of detectors and stored in said storage unit; anintegrating unit configured to integrate the present result from actingby one or more of said plurality of actuators or the present result fromdetecting by one or more of said plurality of detectors and the pastresults from acting by one or more of said plurality of actuators or thepast results from detecting by one or more of said plurality ofdetectors according to a result of said comparing, and outputting as anintegrated result from acting or an integrated result from detecting,wherein the integrated result accounts for past results comprisingattitude parameters that are weighted according to a degree ofexperience of a user operating each of the plurality of actuators, andwherein said plurality of actuators and said plurality of detectorsshare processing through collaboration, whereby said plurality ofactuators of said plurality of detectors are more highly functional ascompared to operating individually.
 2. An information processing systemaccording to claim 1, wherein said plurality of actuators actcorresponding to multiple user operations and predetermined information;and a communication unit receives the state of said plurality ofactuators detected by said plurality of detectors; said system furthercomprising a generator unit configured to generate, as saidpredetermined information, reference information to serve as a referenceregarding the action of said plurality of actuators, based on the stateof said plurality of actuators.
 3. An information processing systemaccording to claim 2, further comprising: a second storage unitconfigured to store said reference information; and an updating unitconfigured to update the stored contents of said second storage unitwith new said reference information generated by said generator unit. 4.An information processing system according to claim 2, wherein saidgenerator unit generates said reference information based on the stateof said plurality of actuators and external information.
 5. Aninformation processing system according to claim 2, further comprising acontrol unit configured to control the actions of one or more of saidplurality of actuators based on said reference information, so that saidone or more actuators have higher functionality than acting alone.
 6. Aninformation processing system according to claim 2, wherein one or moreof said plurality of actuators is a manipulator.
 7. An informationprocessing system according to claim 6, wherein the manipulator is amaster manipulator which controls a slave manipulator which acts upon apredetermined object.
 8. An information processing system according toclaim 7, wherein said reference information is information forrestricting the action directed to said predetermined object.
 9. Aninformation processing system according to claim 7, wherein saidreference information is information for performing surgery upon saidpredetermined object.
 10. An information processing system according toclaim 7, wherein said reference information is information regardingdesign of said predetermined object.
 11. An information processingsystem according to claim 1, wherein the functionality of said pluralityof detectors is increased.
 12. An information processing systemaccording to claim 11, wherein said plurality of actuators are aplurality of moving entities which move according to operations ofmultiple users; wherein said plurality of detectors are provided on saidplurality of moving entities and detect a subject to be imaged; andwherein said communication unit receives output of said plurality ofdetectors; said system further comprising a generation unit configuredto generate information with added value added, by processing the outputof said plurality of detectors.
 13. An information processing systemaccording to claim 12, wherein said detectors are image-taking devicesfor taking images of a subject.
 14. An information processing systemaccording to claim 13, wherein said communication unit receives an imageof said subject output by said image-taking devices; and wherein saidgenerator unit generates an image viewing said subject from multipledirection, which is an image with higher added value as compared to animage obtained from a single image-taking device, by processing theimage of said subject.
 15. An information processing system according toclaim 13, wherein, of said multiple moving entities, one moving entitytransmits to other moving entities a request signal for requestingimages of said subject; and wherein said other moving entities transmitimages of said subject to said one moving entity, in response to saidrequest signal.
 16. An information processing system according to claim15, wherein said one moving entity calculates an image-taking positionfor image-taking of said subject, based on the position of said subject,and the image-taking direction of said subject, and transmits saidrequest signal along with the image-taking position.
 17. An informationprocessing system according to claim 16, wherein said other movingentities take images of said subject at the point that the currentposition thereof matches said image-taking position by said image-takingdevices.
 18. An information processing system according to claim 17,wherein said other moving entities transmit said image-taking positionto said one moving entity along with images taken of said subject; andwherein said one moving entity processes the images of said subject fromsaid other moving entities based on said image-taking positionstransmitted with the images, thereby generating an image wherein saidsubject is viewed from multiple directions.
 19. An informationprocessing system according to claim 13, wherein said image-takingdevices include an all-directional camera capable of taking imagesomni-directionally.
 20. An information processing method comprising thesteps of: performing actions corresponding to operations by a pluralityof users; detecting the real world where a plurality of actuators act;and exchanging information between said plurality of actuators and aplurality of detectors; storing results from acting or results fromdetecting; comparing a present result from acting or a present resultfrom detecting with past results from acting and stored during thestoring step or past results from detecting and stored during thestoring step; integrating the present result from acting or the presentresult from detecting and the past results from acting or the pastresults from detecting according to a result of said comparing; andoutputting as an integrated result from acting or an integrated resultfrom detecting, wherein the integrated result accounts for past resultscomprising attitude parameters that are weighted according to a degreeof experience of a user operating each of the plurality of actuators,and wherein said plurality of actuators and said plurality of detectorsshare processing through collaboration, whereby said plurality ofactuators or said plurality of detectors are more highly functional ascompared to operating individually.
 21. A non-transitory recordingmedium storing a computer-readable program for controlling a computersuch that: a plurality of actuators act corresponding to operations by aplurality of users; a plurality of detectors detect the real world wheresaid plurality of actuators act; and communication units exchangeinformation between said plurality of actuators and said plurality ofdetectors; a storage unit stores results from acting by said pluralityof actuators or results from detecting by said plurality of detectors; acomparator compares a present result from acting by said plurality ofactuators or a present result from detecting by said plurality ofdetectors with past results from acting by said plurality of actuatorsand stored in said storage unit or past results from detecting by saidplurality of detectors and stored in said storage unit; and anintegrator configured to integrate the present result from acting by oneor more of said plurality of actuators or the present result fromdetecting by one or more of said plurality of detectors and the pastresults from acting by one or more of said plurality of actuators or thepast results from detecting by one or more of said plurality ofdetectors according to a result of said comparing, and outputs as anintegrated result from acting or an integrated result from detecting,wherein the integrated result accounts for past results comprisingattitude parameters that are weighted according to a degree ofexperience of a user operating each of the plurality of actuators, andwherein one or more of said plurality of actuators and one or more ofsaid plurality of detectors share processing through collaboration,whereby one or more of said plurality of actuators or one or more ofsaid plurality of detectors are more highly functional as compared tooperating individually.
 22. An information processing device comprising:one or more actuators configured to act corresponding to operations by auser; one or more detectors configured to detect the real world wheresaid one or more actuators acts; a communication unit configured toexchange information between one or more actuators and one or moredetectors; a storage unit configured to store results from acting bysaid one or more actuators or results from detecting by said one or moredetectors; a comparator configured to compare a present result fromacting by the one or more actuators or a present result from detectingby said one or more detectors with past results from acting by one ormore actuators and stored in said storage unit or past results fromdetecting by one or more detectors and stored in said storage unit; andan integrator configured to integrate the present result from acting byone or more actuators or the present result from detecting by one ormore detectors and the past results from acting or the past results fromdetecting by said one or more detectors according to a result of saidcomparing, and outputting as an integrated result from acting or anintegrated result from detecting, wherein the integrated result accountsfor past results comprising attitude parameters that are weightedaccording to a degree of experience of a user operating each of theplurality of actuators, and wherein said one or more actuators and saidone or more detectors share processing with one or more said actuatorsand one or more said detectors through collaboration, whereby said oneor more actuators or said one or more detectors are more highlyfunctional as compared to operating individually.
 23. An informationprocessing device according to claim 22, wherein the functions of saidone or more actuators become highly functional.
 24. An informationprocessing device according to claim 23, wherein said one or moreactuators act corresponding to user operations and predeterminedinformation; and wherein said communication unit receives the state ofsaid one or more actuators detected by said one or more detectors; saiddevice further comprising a generator configured to generate, as saidpredetermined information, reference information to serve as a referenceregarding the action of said one or more actuators, based on the stateof said one or more actuators.
 25. An information processing deviceaccording to claim 24, further comprising: a memory unit configured tostore said reference information; and an updating unit configured toupdate the stored contents of said memory with new said referenceinformation generated by said generator.
 26. An information processingdevice according to claim 24, wherein said generator generates saidreference information based on the state of said one or more actuatorsand external information.
 27. An information processing device accordingto claim 24, further comprising a controller configured to control theactions of the one or more actuators based on said referenceinformation, so that one or more actuators has higher functionality thanacting alone.
 28. An information processing device according to claim24, wherein said at least on of the one or more actuators is amanipulator.
 29. An information processing device according to claim 28,wherein the manipulator is a master manipulator which controls a slavemanipulator which acts upon a predetermined object.
 30. An informationprocessing device according to claim 29, wherein said referenceinformation is information for restricting the action directed to saidpredetermined object.
 31. An information processing device according toclaim 29, wherein said reference information is information forperforming surgery upon said predetermined object.
 32. An informationprocessing device according to claim 29, wherein said referenceinformation is information regarding design of said predeterminedobject.
 33. An information processing device according to claim 22,wherein the functionality of at least one of said one or more detectorsis increased.
 34. An information processing device according to claim33, wherein said one or more actuators include a moving entity whichmoves according to operations of multiple users; wherein said one ormore detectors are provided on said moving entity and detect a subjectto be imaged; and wherein said communication unit receive output of saidat least one of the one or more detectors; said device furthercomprising: a generator configured to generate information with addedvalue added, by processing the output of said at least one detector anda second detector.
 35. An information processing device according toclaim 34, wherein said detectors are image-obtaining units configured toobtain one or more images of a subject.
 36. An information processingdevice according to claim 35, wherein said communication unit receivesan image of said subject output by said image obtaining units; andwherein said generator generates an image that provides viewing of saidsubject from multiple directions, which is an image with higher addedvalue compared to an image obtained from a single image-obtaining unit,by processing the image of said subject.
 37. An information processingdevice according to claim 35, wherein said communication unit transmitsto other moving entities a request signal for requesting images of saidsubject; and receives images of said subject taken by said seconddetector of said other moving entity, in response to said requestsignal.
 38. An information processing device according to claim 37,wherein said communication unit calculates an image-obtaining positionfor obtaining one or more images of said subject, based on the positionof said subject, and the image-taking direction of said subject, andtransmits said request signal along with the image-taking position. 39.An information processing device according to claim 38, wherein, at thepoint that the current position thereof matches said image-obtainingposition, said communication unit receive images of said subject, takenby said image-obtaining unit of other moving entities.
 40. Aninformation processing device according to claim 39, wherein saidcommunication unit processes the images of said subject transmitted fromsaid other moving entities, based on said image-obtaining positionstransmitted with the images, thereby generating an image wherein saidsubject is viewed from multiple directions.
 41. An informationprocessing device according to claim 35, wherein said image-obtainingunit is an omni-directional camera capable of taking imagesomni-directionally.
 42. An information processing method comprising thesteps of: performing an action corresponding to operations by aplurality of users; detecting real world conditions where said actionsoccur, and information is exchanged between one or more actuators andone or more detectors; storing results from acting or results fromdetecting; comparing a present result from acting or a present resultfrom detecting with past results from acting and stored during thestoring step or past results from detecting and stored during thestoring step; integrating the present result from acting or the presentresult from detecting and the past results from acting or the pastresults from detecting according to a result of said comparing; whereinthe integrated result accounts for past results comprising attitudeparameters that are weighted according to a degree of experience of auser operating each of the plurality of actuators, and outputting as anintegrated result from acting or an integrated result from detecting;and sharing processing between a first set of actuators and a first setof detectors and a second set of actuators and a second set of detectorsthrough collaboration, thereby the first set of actuators, first set ofdetectors, second set of actuators and second set of detectors are morehighly functional as compared to operating individually.
 43. Anon-transitory computer-readable medium for storing a computer-readableprogram for controlling a computer such that when executed the programcauses the computer to: act corresponding to operations by a pluralityof users; detect the real world where said acting occurs; and exchangeinformation between actuators and detectors; store results from theactuators or results from the detectors; compare a present result fromthe actuators or a present result from the detectors with past resultsfrom acting by said plurality of the actuators or past results fromdetecting by said plurality of detectors; and integrate the presentresult from acting by said actuators or the present result fromdetecting by said detectors and the past results from acting by saidplurality of actuators or the past results from detecting by saidplurality of detectors according to a result of said comparing, whereinthe integrated result accounts for past results comprising attitudeparameters that are weighted according to a degree of experience of auser operating each of the plurality of actuators; and output as anintegrated result from acting or an integrated result from detecting,wherein said actuators and said detectors share processing throughcollaboration, thereby becoming more highly functional as compared tooperating individually.